Pith. sign in

REVIEW 4 major objections 5 minor 44 references

The paper's central claim is that the attention density a stored memory fragment receives at a middle transformer layer is an inherent, query-aware tracing signal; MemDefrag turns this into a training-free rank-reorder-filter defragmentatio

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-02 08:21 UTC pith:FBBDT7CG

load-bearing objection The headline retention numbers are large and probably real: training-free reorder/filter of latent memory at a middle layer more than doubles step-50 accuracy vs MemoryLLM/M+, but the mechanism story has a few unverified gaps. the 4 major comments →

arxiv 2607.05969 v2 pith:FBBDT7CG submitted 2026-07-07 cs.CL

MemDefrag: Latent Memory Defragmentation for Large Language Models

Ruiyi Yan , Zhuoyuan Mao , Yiwen Guo This is my paper
classification cs.CL
keywords latent memoryattention densitymemory defragmentationknowledge retentiontracing signaltraining-free methodlong-term memorylarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the main obstacle to long-term latent memory in large language models—where past knowledge is stored as per-layer hidden states that degrade as they accumulate—has a training-free fix. The authors find that a narrow band of middle transformer layers reliably concentrates the most attention on the memory fragment a question actually targets, and they turn that observation into a defragmentation step: rank fragments by this attention density at one tracer layer, reorder every layer's memory by that ranking, and keep only the top few fragments for inference. With this, QA accuracy after 50 memory updates rises to 43.0% from about 17.4% for two earlier latent-memory systems, and the method transfers to other models and memory variants without retraining. A proportional forgetting scheme that evicts low-information tokens extends the benefit once capacity is full. A sympathetic reader would care because it suggests latent memory can be made self-organizing without added training or auxiliary modules.

Core claim

The central discovery is that attention density is an inherent tracing signal: when a prompt queries stored memory, a small set of middle transformer layers consistently gives the highest attention density to the target fragment, independent of where that fragment sits in the concatenated memory. In the authors' 32-layer testbed, the best tracer layer ranks the true target first 85.6% of the time (mean rank 1.66) across all 20 possible target positions, and attention density at that layer tracks QA accuracy with correlation exceeding 0.95. MemDefrag operationalizes this by (i) computing attention density at a chosen tracer layer, (ii) sorting fragments by that score and reordering all layers

What carries the argument

The central object is attention density, defined as the mean head-averaged causal attention that the prompt's last token (or all prompt tokens) gives to the positions of a stored memory fragment at a given transformer layer, averaged over the fragment's positions. The tracer layer is the single middle layer whose density ranking best identifies the target fragment—layer 13 in the 32-layer model. This ranking does the work of a query-aware relevance oracle: reordering all layers by it exploits causal recency (placing the target closest to the prompt), and Top-K truncation removes irrelevant fragments and counteracts accumulated positional-encoding distortion. The companion mechanism, informat

Load-bearing premise

The load-bearing premise is that the relevance ordering of memory fragments measured at one middle layer is the correct ordering for every other layer, and that reordering by that single ranking remains valid even though it shifts the position encodings of every fragment—yet the paper never measures whether layers actually agree on that full ordering.

What would settle it

Take a fixed set of stored fragments and queries, then compute the full attention-density ranking of all fragments at every transformer layer. If another layer's top-ranked fragment differs from the tracer layer's top-ranked fragment in more than a small fraction of queries—or if reordering each layer by its own density ranking beats reordering every layer by the single tracer-layer ranking on a knowledge-retention benchmark—the paper's central claim about layer-invariance would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If one middle layer's attention density reliably traces the target fragment, latent-memory systems can be managed without co-trained retrievers or continual training, making them plug-and-play upgrades to existing models.
  • Reordering fragments by density alone—without any filtering—already slows the retention decline, indicating that causal recency is a major component of the gain.
  • The optimal number of retained fragments stays small (Top-1 to Top-4) and does not grow with the number of updates, suggesting that concentrating attention density is more valuable than widening the candidate pool.
  • Across four tested instruction-tuned models, the best tracer layer falls consistently in the [L/3, L/2] band of layer depth, so deploying on a new backbone requires only a narrow sweep over a few candidate layers.
  • The framework composes with prompt compression: compressing knowledge fragments makes memory grow more slowly, delays the onset of forgetting, and improves retention at late update steps.
  • The paper itself notes that its Top-K choice is static and that on one model Top-1 filtering degrades after roughly 20 updates, which directly motivates adaptive selection of K from the density distribution.
  • A natural testable extension is to apply the same attention-density ranking to ordinary long-context inference without latent memory, effectively turning the tracer layer into a training-free retrieval oracle.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The single-tracer-layer design assumes the fragment ordering measured at one middle layer is the correct ordering for every other layer; testing cross-layer agreement of the full ranking (not just the top target) would either validate this or reveal the need for per-layer or ensemble orderings.
  • The paper reports static Top-K as a limitation and notes a specific model where Top-1 drifts after ~20 steps; an adaptive K chosen from the shape of the per-fragment density distribution is the most direct next step and aligns with the paper's own suggestion.
  • Because the forgetting mechanism uses token-level self-information, one could test whether adaptive per-fragment quotas—rather than length-proportional quotas—further improve retention when fragments contain unequal amounts of relevant information.
  • The tracing signal's compatibility with prompt compression implies a broader principle: attention density at the tracer layer can serve as a cheap, training-free salience measure for deciding what to keep, compress, or discard in any memory system.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses long-term latent memory in LLMs, where per-layer hidden states of past knowledge fragments are concatenated into a persistent prefix. It first demonstrates that QA accuracy degrades sharply as this memory grows, and shows that the attention density received by the target fragment at middle layers (especially layer 13 in Llama-3.1-8B-Instruct) correlates strongly with accuracy and can identify the target fragment among many stored fragments with high Top-1 accuracy. Building on this, the paper proposes MemDefrag, a training-free, model-agnostic framework that (i) ranks memory fragments by attention density at a selected tracer layer, reorders them by this ranking, and keeps only the Top-K fragments at inference time, and (ii) uses a self-information-based proportional forgetting strategy when memory capacity is exceeded. Experiments on NaturalQA and SQuAD for 50 memory updates and on six LongBench datasets report substantial gains over MemoryLLM and M+, with generalization across four LLMs and compatibility with additional latent-memory variants and prompt compression.

Significance. If the central claims hold, MemDefrag is a noteworthy contribution: it is simple, training-free, requires no auxiliary modules, and turns an internal attention signal into a practical retrieval mechanism for latent memory. The paper has clear strengths: the code is released; the evaluation protocol removes positional bias by shuffling fragments; a no-memory borderline is reported; the no-positional-distortion control in Appendix C helps isolate the effect of positional misalignment; and the method is tested on multiple LLMs and with two attention-aggregation strategies. The reported retention gains are large (e.g., 43.0% vs. 17.4% at step 50 on NaturalQA). However, the mechanism rests on an unverified layer-invariance assumption, and the comparison with baselines is not budget-matched; both issues are load-bearing for the paper's headline claims.

major comments (4)
  1. [§4.1, Algorithm 1 (Eqs. 6–9)] The defragmentation ranking π is computed from attention density at a single tracer layer l*, then applied verbatim to reorder and truncate memory at every layer. The paper validates that layer 13 can identify the target fragment at that same layer (Table 1: mean rank 1.66, Top-1 85.6%), but it never measures whether the full fragment ordering at l* agrees with the orderings at other layers. Tables 4–7 show that other layers have materially different tracing accuracy; for small K, if another layer's most relevant fragment differs from l*'s top fragment, that layer's retained memory may be missing the information it needs. This is the central mechanistic premise of MemDefrag and remains unverified. Please report, on the same evaluation data, the agreement between the ranking at l* and the ranking at every other layer (e.g., Kendall's tau or Spearman correlation), and/or compare retention
  2. [§5.1.3, Table 2; §5.2, Table 3] The comparison with MemoryLLM and M+ is budget-asymmetric. As the paper states, MemoryLLM and M+ compress each 512-token chunk into 256 memory vectors per layer, while MemDefrag stores the full 512 hidden states per layer for each fragment. With Nmax equal to 12,800 in all cases, MemDefrag retains twice as many raw states per knowledge item before the forgetting mechanism is triggered, and in the 50-step retention experiment it experiences no capacity pressure until step ~25 while the baselines compress from the first update. The headline '43.0% vs. 17.4%' may therefore reflect memory capacity rather than defragmentation. Please add a budget-matched ablation, e.g., compress MemDefrag's fragments to 256 states per layer (by pooling or by the same compressor used by the baselines), or set Nmax so that the number of fragments retained is equal. Similarly, Table 3 uses passage-based/256-toke
  3. [§5, experiments setup] The base model used for the three compared systems is stated ambiguously. Section 5 says both that all comparative experiments are run on Llama-3.1-8B-Instruct and that 'all three models are built upon Llama-3.1-8B', while the baseline checkpoints are named Memoryllm-8b and M+-8b. If the baselines are based on the non-instruction-tuned Llama-3.1-8B while MemDefrag uses Llama-3.1-8B-Instruct, the comparison is confounded by instruction tuning. Please state the exact base checkpoint for each system and, if necessary, run the baselines on Llama-3.1-8B-Instruct or run MemDefrag on Llama-3.1-8B.
  4. [Tables 1–3, Figures 4–5] No error bars, confidence intervals, or significance tests are reported anywhere, despite the fact that many LongBench differences are small (e.g., 2WikiMultihopQA string match 36.00 vs. 35.60; F1 9.51 vs. 9.41). The retention claims at step 50 involve large gaps, but the paper should still report variance over the 500 evaluation groups, at least for the headline comparisons, so the reader can judge whether the reported advantages are stable.
minor comments (5)
  1. [Algorithm 1] The assignment 'θ^p_n ← {m_{π(n−K+1)}, ..., m_{π(n)}}' is inside the 'for each layer' loop, although it defines the output once. Move it outside the loop or clarify that it is the same object for all layers.
  2. [Table 12] Header 'Ration' should be 'Ratio'.
  3. [§5.1.3] The statement 'each 512-token chunk is compressed into 256 memory vectors per layer in a fixed 2:1 ratio' is clear, but it is worth stating explicitly in the main text that MemDefrag stores the uncompressed hidden states, since this is the source of the budget asymmetry discussed above.
  4. [Appendix E.4 / Table 10] The claim 'Top-2 tracing accuracy generally exceeds 90%' is true for the selected top layer, but Table 10 shows several entries below 90% (e.g., Qwen2.5 NaturalQA 90.3 is barely above; Llama-3.1 SQuAD 86.1). Please phrase accurately, e.g., 'is usually above 86% and often above 90%'.
  5. [§4.2.2 / Eq. (16)-(17)] The self-information I(x_i^(t)) is computed from the base LLM's token probabilities. It would help to state explicitly that these probabilities are computed on the original text fragment before the fragment is stored, since the notation 'p_φ' could be mistaken for the memory-augmented model.

Circularity Check

0 steps flagged

No significant circularity: the tracing signal and retention gains are measured quantities, not fitted inputs.

full rationale

The central derivation is empirical rather than definitional. Attention density ρ_i^l is defined in Eq. 5 as a measured head-averaged attention score from the prompt to each stored fragment, and Eqs. 6–9 / Algorithm 1 apply an ordering computed from that measurement and then evaluate QA accuracy as a separate observable. The tracing evaluation in Table 1 checks whether the measured density ranks a dataset-known target first; it does not define the target in terms of the density ranking used at inference. Layer 13, Top-K, and the last-token vs all-token variant are selected empirically (Tables 1 and 4), but the reported retention numbers are not forced by that selection: selecting a hyperparameter on a benchmark is a validity/selection concern, not an equation-level identity. The only author self-citation (Li et al. 2026b, which includes author Yiwen Guo) appears in a general Related Work sentence about LLM memory and agents and is not load-bearing. The Limitations section's admission that Top-K is chosen empirically and held static is a limitation statement, not evidence of circularity. The skeptic's concerns about cross-layer transfer of the fragment ranking and stability under reordering are unverified assumptions and correctness risks, but the paper never makes the target definition or the reported accuracy equal to the density ranking by construction. The method is self-contained and compared against external baselines and benchmarks, so no significant circularity is found.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 0 invented entities

No new physical or architectural entities are introduced: attention density rho is a derived observable from existing attention. The central claim rests on domain assumptions inherited from the latent-memory paradigm (validity of concatenated hidden states, attention density as relevance proxy, self-information as importance), plus five method-level free parameters (K, tracer layer, Nmax, attention variant, chunking). The most fragile entries are the layer-invariance/ranking-stability assumption (ad_hoc_to_paper) and the empirically-grounded-but-in-sample choices of K and the tracer layer.

free parameters (5)
  • Top-K filtering number = K=2 for most experiments; K=1 best at step 50 on NaturalQA
    Limitations: 'Top-K filtering is chosen empirically and held static.' K in {1,2,3,4,5,6,all} swept on the same benchmarks; adaptive K is explicitly left for future work.
  • Tracer layer l* = Layer 13/32 for Llama-3.1-8B-Instruct; 14/28 Qwen2.5-7B; 15/32 Mistral-7B; 15/42 Gemma-2-9b
    Selected by the tracing-capability study in Section 3 (Table 1) run on the same NaturalQA/SQuAD benchmarks; the paper recommends a [L/3, L/2] band sweep for new backbones.
  • Maximum memory capacity Nmax = 12,800 hidden states
    Set empirically equal to the memory-token count of MemoryLLM/M+ (Section 5); determines the step (26) at which forgetting begins.
  • Attention computation strategy = last-token for NaturalQA; all-token for SQuAD
    Chosen per dataset based on tracing-capability rankings in Table 4; strategy choice is part of the method configuration and affects reported accuracy.
  • Long-context chunking strategy = passage-based for 2WikiMultihopQA/HotpotQA/MuSiQue; >=256-token sentence-bounded chunks for MultiFieldQA/NarrativeQA/Qas
    Chosen per dataset in Appendix E.2; different chunking changes fragment granularity and therefore the defragmentation behavior.
axioms (7)
  • domain assumption Vanilla latent-memory update rule: concatenating per-layer hidden states formed by a training-free prefill is a valid way to evolve memory without retraining (Eqs. 1-3).
    Inherited from MemoryLLM/M+; the entire evaluation runs inside this paradigm, and MemDefrag is presented as an extension of it.
  • domain assumption Positional-encoding misalignment is the dominant cause of retention collapse (Figure 2a vs Appendix C.1).
    Used to motivate the method; Appendix C.1 shows accuracy is more stable without latent memory, but latent-representation drift is a confounder in that comparison.
  • domain assumption Attention density at the tracer layer is an adequate proxy for the fragment relevance that matters for the final answer.
    Established empirically in Section 3 (Top-1 85.6% on NaturalQA) but not proven for all prompts; the whole filtering step depends on this proxy.
  • domain assumption Token self-information -log p(t) is a measure of a memory token's importance for future retrieval (Eq. 16).
    Borrowed from prompt-compression literature (Selective Context, LLMLingua); the only validation here is the random-forgetting ablation (Figure 7).
  • ad hoc to paper The fragment relevance ordering measured at one tracer layer on the original concatenation remains correct for all layers and after reordering (Eqs. 6-9).
    Rho is computed on the original prefix ordering, then the memory is reordered and used for the final forward pass; ranking stability across layers and across reorderings is never measured.
  • domain assumption String-match accuracy (gold answer in generated text) is a sufficient proxy for retention quality.
    Used for all knowledge-retention numbers; standard among the baselines but coarse.
  • standard math Causal-masked multi-head self-attention with softmax and the linear-algebra derivation of the head-averaged attention matrix (Appendix B.1).
    Background for Eqs. 12-13; standard transformer mechanics.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of MemDefrag: Latent Memory Defragmentation for Large Language Models." pith.science (2026). https://pith.science/paper/FBBDT7CG

@misc{pith2026260705969,
  author       = {Pith},
  title        = {Pith review of: MemDefrag: Latent Memory Defragmentation for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FBBDT7CG}},
  note         = {Machine review of arXiv:2607.05969}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Latent memory, which stores past knowledge fragments as per-layer hidden states, has emerged as a promising paradigm (e.g., MemoryLLM and M+) for long-term memory in large language models (LLMs). However, the paradigm suffers from significant performance degradation during memory updates, due to positional encoding misalignment and the absence of any tracing mechanism to distinguish target memory fragments from irrelevant ones. To discover such a tracing mechanism, we probe the layer-wise attention density over stored memory fragments, and find that a small set of middle transformer layers consistently concentrates the highest density on the target fragment - exposing an inherent tracing signal. In light of this, we propose MemDefrag, a training-free and model-agnostic framework that (1) uses a middle-layer tracing signal to conduct memory defragmentation (rank, reorder, and filter memories), and (2) applies an informativeness-guided proportional forgetting mechanism once capacity is exceeded. Experiments show that MemDefrag substantially outperforms MemoryLLM and M+ on knowledge retention (e.g., 43.0% vs. 17.4%/17.6% after 50 memory updates) and long-context benchmarks, and generalizes well across various LLMs and latent-memory variants. The code is available at github.com/ryehr/MemDefrag.

Figures

Figures reproduced from arXiv: 2607.05969 by Ruiyi Yan, Yiwen Guo, Zhuoyuan Mao.

Figure 1
Figure 1. Figure 1: Memory formation, evolution and model in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Effects and analyses of positional distortion [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An example for the defragmentation processes of our MemDefrag framework, in which the time step is 8 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: 20-step knowledge retention on NaturalQA [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: 50-step knowledge retention on NaturalQA [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effects and analyses attention density without [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: 50-step knowledge retention comparing informativeness-based vs. random forgetting on Natu￾ralQA and SQuAD, of which Top-2 or Top-3 highest￾ranked memory fragments are filtered of defragmenta￾tion. their gap increases over time. Specifically, on Nat￾uralQA with Top-2, informativeness-based forget￾ting sustains an accuracy of 37.8% at step 50, com￾pared to 20.8% for random forgetting, indicating an improveme… view at source ↗
Figure 8
Figure 8. Figure 8: 50-step knowledge retention comparing MemDefrag (with Top-1, 2, 3 or 4 filtering) vs. vanilla latent memory on NaturalQA and SQuAD. (2) Similar to experiment results ( [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: 50-step knowledge retention on NaturalQA [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Proceedings of the 42nd International Conference on Machine Learning , pages =

    Layer by Layer: Uncovering Hidden Representations in Language Models , author =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , editor =

    2025

  3. [3]

    2024 , url=

    Jiaming Tang and Yilong Zhao and Kan Zhu and Guangxuan Xiao and Baris Kasikci and Song Han , booktitle=. 2024 , url=

    2024

  4. [4]

    2025 , eprint=

    Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory , author=. 2025 , eprint=

    2025

  5. [5]

    Neuron , volume=

    Key-value memory in the brain , author=. Neuron , volume=. 2025 , publisher=

    2025

  6. [6]

    Memory ^3 : Language Modeling with Explicit Memory , volume=

    Yang, Hongkang and Lin, Zehao and Wang, Wenjin and Wu, Hao and Li, Zhiyu and Tang, Bo and Wei, Wenqiang and Wang, Jinbo and Tang, Zeyun and Song, Shichao and Xi, Chenyang and Yu, Yu and Chen, Kai and Xiong, Feiyu and Tang, Linpeng and E, Weinan , year=. Memory ^3 : Language Modeling with Explicit Memory , volume=. Journal of Machine Learning , publisher=....

    doi:10.4208/jml.240708

  7. [7]

    Proceedings of the 41st International Conference on Machine Learning , articleno =

    Wang, Yu and Gao, Yifan and Chen, Xiusi and Jiang, Haoming and Li, Shiyang and Yang, Jingfeng and Yin, Qingyu and Li, Zheng and Li, Xian and Yin, Bing and Shang, Jingbo and McAuley, Julian , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

    2024

  8. [8]

    International Conference on Learning Representations , year=

    Memorizing Transformers , author=. International Conference on Learning Representations , year=

  9. [9]

    Rusu and Kieran Milan and John Quan and Tiago Ramalho and Agnieszka Grabska-Barwinska and Demis Hassabis and Claudia Clopath and Dharshan Kumaran and Raia Hadsell , title =

    James Kirkpatrick and Razvan Pascanu and Neil Rabinowitz and Joel Veness and Guillaume Desjardins and Andrei A. Rusu and Kieran Milan and John Quan and Tiago Ramalho and Agnieszka Grabska-Barwinska and Demis Hassabis and Claudia Clopath and Dharshan Kumaran and Raia Hadsell , title =. Proceedings of the National Academy of Sciences , volume =. 2017 , doi ...

    doi:10.1073/pnas.1611835114 2017

  10. [10]

    Edward J Hu and yelong shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle=. Lo. 2022 , url=

    2022

  11. [11]

    C haracter GLM : Customizing Social Characters with Large Language Models

    Zhou, Jinfeng and Chen, Zhuang and Wan, Dazhen and Wen, Bosi and Song, Yi and Yu, Jifan and Huang, Yongkang and Ke, Pei and Bi, Guanqun and Peng, Libiao and Yang, JiaMing and Xiao, Xiyao and Sabour, Sahand and Zhang, Xiaohan and Hou, Wenjing and Zhang, Yijia and Dong, Yuxiao and Wang, Hongning and Tang, Jie and Huang, Minlie. C haracter GLM : Customizing ...

    2024

  12. [12]

    2025 , eprint=

    Pre-training Limited Memory Language Models with Internal and External Knowledge , author=. 2025 , eprint=

    2025

  13. [13]

    M+: Extending Memory

    Yu Wang and Dmitry Krotov and Yuanzhe Hu and Yifan Gao and Wangchunshu Zhou and Julian McAuley and Dan Gutfreund and Rogerio Feris and Zexue He , booktitle=. M+: Extending Memory. 2025 , url=

    2025

  14. [14]

    and Lin, Kevin and Hewitt, John and Paranjape, Ashwin and Bevilacqua, Michele and Petroni, Fabio and Liang, Percy

    Liu, Nelson F. and Lin, Kevin and Hewitt, John and Paranjape, Ashwin and Bevilacqua, Michele and Petroni, Fabio and Liang, Percy. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. 2024. doi:10.1162/tacl_a_00638

    doi:10.1162/tacl_a_00638 2024

  15. [15]

    2024 , eprint=

    MemGPT: Towards LLMs as Operating Systems , author=. 2024 , eprint=

    2024

  16. [16]

    2024 USENIX Annual Technical Conference (USENIX ATC 24) , year =

    Bin Gao and Zhuomin He and Puru Sharma and Qingxuan Kang and Djordje Jevdjic and Junbo Deng and Xingkun Yang and Zhou Yu and Pengfei Zuo , title =. 2024 USENIX Annual Technical Conference (USENIX ATC 24) , year =

    2024

  17. [17]

    2025 , eprint=

    SeqPE: Transformer with Sequential Position Encoding , author=. 2025 , eprint=

    2025

  18. [18]

    2026 , eprint=

    RePo: Language Models with Context Re-Positioning , author=. 2026 , eprint=

    2026

  19. [19]

    2026 , eprint=

    Time is Not a Label: Continuous Phase Rotation for Temporal Knowledge Graphs and Agentic Memory , author=. 2026 , eprint=

    2026

  20. [20]

    International Conference on Learning Representations , year=

    BERTScore: Evaluating Text Generation with BERT , author=. International Conference on Learning Representations , year=

  21. [21]

    Gemma , url=

    Gemma Team , year=. Gemma , url=. doi:10.34740/KAGGLE/M/3301 , publisher=

    doi:10.34740/kaggle/m/3301

  22. [22]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    2023

  23. [23]

    Qwen2.5: A Party of Foundation Models , url =

    Qwen Team , month =. Qwen2.5: A Party of Foundation Models , url =

  24. [24]

    ROUGE : A Package for Automatic Evaluation of Summaries

    Lin, Chin-Yew. ROUGE : A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out. 2004

    2004

  25. [25]

    L ong B ench: A Bilingual, Multitask Benchmark for Long Context Understanding

    Bai, Yushi and Lv, Xin and Zhang, Jiajie and Lyu, Hongchang and Tang, Jiankai and Huang, Zhidian and Du, Zhengxiao and Liu, Xiao and Zeng, Aohan and Hou, Lei and Dong, Yuxiao and Tang, Jie and Li, Juanzi. L ong B ench: A Bilingual, Multitask Benchmark for Long Context Understanding. Proceedings of the 62nd Annual Meeting of the Association for Computation...

    doi:10.18653/v1/2024.acl-long.172 2024

  26. [26]

    LLML ingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression

    Pan, Zhuoshi and Wu, Qianhui and Jiang, Huiqiang and Xia, Menglin and Luo, Xufang and Zhang, Jue and Lin, Qingwei and R. LLML ingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.findings-acl.57

    doi:10.18653/v1/2024.findings-acl.57 2024

  27. [27]

    LLML ingua: Compressing Prompts for Accelerated Inference of Large Language Models

    Jiang, Huiqiang and Wu, Qianhui and Lin, Chin-Yew and Yang, Yuqing and Qiu, Lili. LLML ingua: Compressing Prompts for Accelerated Inference of Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.825

    doi:10.18653/v1/2023.emnlp-main.825 2023

  28. [28]

    2024 , eprint=

    Steering Language Models With Activation Engineering , author=. 2024 , eprint=

    2024

  29. [29]

    2025 , eprint=

    Representation Engineering: A Top-Down Approach to AI Transparency , author=. 2025 , eprint=

    2025

  30. [30]

    Yuhong Li and Yingbing Huang and Bowen Yang and Bharat Venkitesh and Acyr Locatelli and Hanchen Ye and Tianle Cai and Patrick Lewis and Deming Chen , booktitle=. Snap. 2024 , url=

    2024

  31. [31]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  32. [32]

    SQ u AD : 100,000+ Questions for Machine Comprehension of Text

    Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy. SQ u AD : 100,000+ Questions for Machine Comprehension of Text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016. doi:10.18653/v1/D16-1264

    doi:10.18653/v1/d16-1264 2016

  33. [33]

    2024 , eprint=

    Steering Llama 2 via Contrastive Activation Addition , author=. 2024 , eprint=

    2024

  34. [34]

    2026 , eprint=

    NextMem: Towards Latent Factual Memory for LLM-based Agents , author=. 2026 , eprint=

    2026

  35. [35]

    and Uszkoreit, Jakob and Le, Quoc and Petrov, Slav

    Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and Toutanova, Kristina and Jones, Llion and Kelcey, Matthew and Chang, Ming-Wei and Dai, Andrew M. and Uszkoreit, Jakob and Le, Quoc and Petrov, Slav. Natura...

    doi:10.1162/tacl 2019

  36. [36]

    2022 , eprint=

    The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink , author=. 2022 , eprint=

    2022

  37. [37]

    Compressing Context to Enhance Inference Efficiency of Large Language Models

    Li, Yucheng and Dong, Bo and Guerin, Frank and Lin, Chenghua. Compressing Context to Enhance Inference Efficiency of Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.391

    doi:10.18653/v1/2023.emnlp-main.391 2023

  38. [38]

    2026 , eprint=

    Memory in the Age of AI Agents , author=. 2026 , eprint=

    2026

  39. [39]

    Frontiers of Computer Science , volume=

    Tool learning with large language models: A survey , author=. Frontiers of Computer Science , volume=. 2025 , publisher=

    2025

  40. [40]

    ACM Trans

    Zhang, Zeyu and Dai, Quanyu and Bo, Xiaohe and Ma, Chen and Li, Rui and Chen, Xu and Zhu, Jieming and Dong, Zhenhua and Wen, Ji-Rong , title =. ACM Trans. Inf. Syst. , month = sep, articleno =. 2025 , issue_date =. doi:10.1145/3748302 , abstract =

    doi:10.1145/3748302 2025

  41. [41]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  42. [42]

    Dan Gusfield , title =. 1997

    1997

  43. [43]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  44. [44]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.