Pith. sign in

REVIEW 4 major objections 2 minor 60 references

A handful of hidden states from a frozen LLM can act as a reusable memory signal, lifting long-context reasoning by 10–29 F1 points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:57 UTC pith:4I72H4X4

load-bearing objection A solid, clearly-written method paper with genuine cross-domain gains, but the central causal claim about hidden-state memory needs one control experiment and multi-seed reporting before I'd trust the headline numbers. the 4 major comments →

arxiv 2607.29032 v1 pith:4I72H4X4 submitted 2026-07-31 cs.MA cs.CL

TransMem: Transforming Hidden States into Memory for Large Language Models

Haodong Lei , Junming Liu , Yirong Chen , Pinlong Cai , Botian Shi , Ding Wang , Hongsong Wang This is my paper
classification cs.MA cs.CL
keywords transformer hidden statesparametric memorylong-context reasoningself-distillationevidence-conditioned trainingLLM agentsmemory modulegating network
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

TransMem claims that a few strategically selected hidden states — the final token of each segment of a long context — carry enough query-relevant information that a learned transformation can turn them into a memory signal, improving an LLM's long-context reasoning. The method inserts a small gated module into the last layers of a frozen backbone; during answer generation, it computes a memory shift from these sparse states and adds it to the current hidden state. Training uses evidence-conditioned self-distillation: a student that sees the full context is trained to match the predictions of a teacher that sees only the gold evidence, so the module learns to preserve evidence and suppress noise. The paper reports consistent gains across three benchmarks (LoCoMo, HotpotQA, MemoryAgentBench) and three backbone architectures and scales, with the module's computational cost independent of context length. If true, this suggests long-context memory can be improved by exploiting representations already computed at prefill time, without scaling the backbone or storing external text.

Core claim

The central claim is that sparse historical hidden states, specifically the final hidden state of each context segment, retain enough query-relevant information to recover the predictive distribution that would be obtained if the model saw only the gold evidence. TransMem learns to transform these segment-final states into a memory shift via a lightweight transformer block, a linear projection, and a learned gate, and applies this shift to the current hidden state during answer generation. The paper demonstrates this on LoCoMo, HotpotQA, and MemoryAgentBench, with F1 gains of 11.58–29.25, 10.20–13.03, and an average accuracy increase from 29.54% to 40.00% across Qwen3-4B, Qwen2.5-14B, and Ll

What carries the argument

The central object is the TransMem module, consisting of a single transformer block, a learnable projection, and a gating network, inserted into the last four layers of a frozen backbone. At inference, the long context is divided into B segments (default B=4); the final hidden state of each segment is aggregated and passed through the module to produce a memory shift, which is scaled by a gate and added to the current generation hidden state. The gate learns a layer-specific amplification-to-suppression schedule (strong in the deepest layer, progressively weaker upward) that transfers across domains and backbones, enabling context-length-independent computation.

Load-bearing premise

The method assumes that the final hidden state of each of a few context segments retains enough query-relevant information from the entire segment that a learned memory shift can recover the evidence-conditioned prediction.

What would settle it

A controlled experiment on a long-context QA dataset where the evidence is deliberately placed at the beginning of each segment (far from the segment-final token), while keeping segment size fixed, should cause a sharp performance drop if the segment-final state is the only information source; alternatively, an ablation that replaces the segment-final state with a random-position hidden state or a mean-pooled representation would reveal whether the choice of summary is critical to the method's success.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Training the memory module on one long-context benchmark (HotpotQA) transfers to other benchmarks with different context distributions (LoCoMo, MemoryAgentBench), suggesting the module learns a generalizable memory capability.
  • Since the module's computation depends only on the number of segments, not the context length, it can scale to contexts longer than the backbone's window by chunking the input and concatenating segment memories.
  • The decoupling of memory capability from backbone reasoning capacity implies that future LLMs could allocate more parameters to reasoning while relying on lightweight memory modules for retention, potentially reducing the need for larger backbones.
  • The evidence-conditioned self-distillation objective could serve as a template for training other latent memory designs, providing a softer, token-level supervision signal that outperforms answer-level reinforcement learning in this setup.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct testable consequence is that the method should be robust to evidence location within a segment; if placing the evidence near the start of a long segment (far from the segment-final token) causes a sharp performance drop, it would indicate the segment-final state is too lossy and performance is bounded by segment granularity.
  • The layer-specific gate schedule hints that the memory shift may be functioning primarily as a re-weighting of deeper-layer representations; investigating the gradient of the gate with respect to evidence position could reveal which layers carry the most critical evidence and whether the schedule can be simplified.
  • Because the teacher sees only the gold evidence, the student must reconstruct the evidence-conditioned prediction from a lossy summary; one could extend the method to iterative memory refinement, letting the module re-process its own output to recover more evidence than a single pass allows.
  • The method's reliance on hidden states at segment boundaries suggests it may be complementary to positional-bias mitigation techniques, which could be combined to further improve long-context utilization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 2 minor

Summary. The paper proposes TransMem, a lightweight parametric memory module for frozen LLMs. During inference, TransMem selects segment-final hidden states from the prefill as a sparse memory H_M (Eqs. 4–5), processes them with a trainable transformer block, projection, and gating network, and adds the resulting residual to the hidden states of the last four layers during answer generation (Eqs. 7–8). The module is trained by evidence-conditioned self-distillation (Eq. 3): a student with full context matches the predictive distribution of a teacher that sees only gold evidence and the question, using the same frozen backbone. Experiments on LoCoMo, HotpotQA, and MemoryAgentBench report consistent gains over baseline backbones and prior memory methods, cross-domain transfer from HotpotQA-only training, and context-independent inference overhead. The paper includes ablations of module components, insertion depth, training objective, gate parameterization, and memory-segment sensitivity.

Significance. If the causal role of historical hidden states were established, this would be a useful contribution: it shows that a small trainable module can improve long-context utilization of a frozen LLM, with sparse memory representations and cross-domain transfer. The self-distillation setup is a genuine strength: the teacher signal is a behavior, not a fitted constant, and teacher and student share the same frozen backbone. The paper also ships code and provides several ablations and efficiency measurements. However, the current experiments do not isolate whether the gains are due to the content of the historical hidden states, and architecture decisions are made on a benchmark that is also used for the main claims. These gaps need to be closed for the paper's central claim to be supported.

major comments (4)
  1. [Inference with TransMem; Table 3] The central claim that sparse historical hidden states are the memory substrate is not separated from the effect of adding a trainable module. The only ablation that removes the memory content is 'w/o Transformer Block' (Table 3), which removes the entire module. A module that transforms H_A alone, or applies a generic correction, would give the same ablation pattern. A direct control is needed: replace H_M with random vectors of the same shape (or with hidden states from an unrelated context), keeping the module trainable. If performance does not collapse, the gains cannot be attributed to the content of H_M. Table 13 strengthens this concern: varying B from 4 to 32 changes scores by at most 1.0 point, suggesting low sensitivity to which segment-final states are used.
  2. [Insertion Depth and Gate Parameterization; Tables 4, 7] The final architecture (Last-4 layers; g=2σ(z) gate) is selected by LoCoMo F1, and LoCoMo is also one of the three main test benchmarks. This is test-set selection: the reported LoCoMo numbers are the best of several configurations tried on that benchmark, which inflates the cross-domain generalization claim. Please choose hyperparameters on a held-out set (e.g., LongMemEval or a HotpotQA validation split) and report LoCoMo only after that choice is fixed; otherwise the LoCoMo results should be presented as a model-selection table, not as an unbiased evaluation.
  3. [Experimental Setup] All experiments use a fixed random seed of 42, and the paper reports no variance or significance information. Because several ablated configurations differ by less than 0.5 F1 (e.g., Tables 7 and 10), single runs cannot distinguish real effects from training noise. The main tables (Tables 1, 2, and the key ablations) should report means and standard deviations over at least three seeds, or the fine-grained comparative claims should be softened.
  4. [Inference with TransMem; Further Study; Figure 3] The claim that TransMem's computational cost is independent of context length is stated without the qualification needed for chunked inputs. The method section describes chunk-wise extraction when C exceeds the backbone window, producing H_M = Concat(H^1_M,...,H^S_M). For such inputs |H_M| = S·B, so the TransMem overhead grows with the number of chunks S. The constant-overhead statement is valid only within a single chunk/context window; please add this caveat to Figure 3 and the efficiency discussion.
minor comments (2)
  1. [Inference with TransMem] The number of memory states is inconsistent. The text says 'the optimal value being only 5', but the default and the checkpoint used in the sensitivity experiments is B=4 (Table 13, 'retain B=4 as default'). Clarify whether the main results use B=4 or B=5.
  2. [Appendix, Table 8] Table 8 reports Exact Match/Contains on an archived 128-example split, while the main HotpotQA results use EM/F1 on the full test protocol. The relationship between these two evaluation settings should be explained in the main text so readers understand which numbers are directly comparable.

Circularity Check

0 steps flagged

No significant circularity: TransMem's empirical claims are supported by held-out benchmarks and standard distillation; the few self-citations are not load-bearing.

full rationale

The paper's central derivation is the evidence-conditioned self-distillation objective (Eq. 3): a memory-augmented student with full context C matches the predictive distribution of an evidence-only teacher with evidence E and query Q, both sharing the same frozen backbone. This is a standard privileged-information distillation setup. The teacher's distribution is not constructed from the student's parameters or from the evaluation metrics; the student is trained on HotpotQA and evaluated on held-out LoCoMo, HotpotQA, and MemoryAgentBench. The benchmark gains are therefore not forced by construction. The inference correction (Eqs. 4-7) is a learned function of historical hidden states HM; no parameter is fitted directly to a target that equals the reported prediction. The claim that sparse historical hidden states are an effective memory substrate is an empirical hypothesis supported by experiments, not an analytical consequence of the definitions. The absence of a control that randomizes HM is an experimental-validity concern (the module could in principle rely on generic corrections rather than content from HM), but this is not a circularity: the reported F1 values are not definitionally equal to the training loss or to a fitted parameter. Minor self-citations (MemCoT, MemVerse) appear only in related-work summaries and do not provide load-bearing justification; no uniqueness theorem or ansatz is imported from the authors' own prior work. The appendix candidly flags non-controlled comparisons and a small evaluation-overlap issue, which further indicates transparency rather than circular reasoning. Overall, the derivation chain is self-contained with respect to its empirical claims; the score reflects only the presence of minor non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The paper's central claim rests on the choice of hyperparameters (number of segments, injection depth, gate temperature/scale) and on several domain assumptions about the informativeness of segment-final hidden states and the transferability of evidence-conditioned distillation. No external benchmarks or formal derivations are used to justify these assumptions beyond the reported experiments.

free parameters (4)
  • Number of memory segments B = 4 (default; main text says optimal is 5)
    Chosen by hand. The paper claims 'optimal value being only 5' but uses B=4 in the main experiments and sensitivity analysis (Table 13). The number of memory slots controls the sparsity of the memory representation and is a key design choice.
  • Number of injected layers (Last-4) = 4 (last four layers)
    Selected based on LoCoMo F1 in the Insertion Depth ablation (Table 4). This hyperparameter is tuned on the same benchmark used for the headline result, which risks selection bias.
  • Gate temperature τ = Not specified
    The gating network uses temperature τ in Eq. (6), but the paper never gives its value or a sensitivity analysis. This is a free hyperparameter that affects the gate behavior.
  • Gate scale α = 2
    Fixed to 2 in Eq. (6); the appendix tests an alternative 3σ(z)−1, but α is not justified beyond the ablation.
axioms (4)
  • domain assumption Each hidden state in a causal transformer aggregates information from all preceding tokens, so the final hidden state of a segment can summarize that segment.
    Figure 1(a) and the paragraph in 'Introduction' rely on this to use h_K_n, ..., h_K_L as memory features. It is a known property of causal transformers, but the sufficiency for question answering is assumed.
  • ad hoc to paper The evidence-only teacher's predictive distribution is a reliable supervisory signal; minimizing forward KL to it teaches transferable memory utilization.
    Eq. (3) defines the training objective. This is a design choice specific to this paper, and it assumes that the teacher distribution is both reachable by the student and beneficial when generalized.
  • domain assumption Chunk-wise memory extraction, where each chunk is encoded independently and memories are concatenated, preserves cross-chunk information.
    In 'Inference with TransMem', when C exceeds the context window, chunks are processed separately and H_M = Concat(H^1_M, ..., H^S_M). This assumes that concatenating per-chunk memories is sufficient without joint encoding.
  • standard math The backbone LLM and its KV cache can provide access to arbitrary historical hidden states without additional forward passes.
    The method relies on reading h_K_t from the prefill output. This is standard inference-time access, but the implementation must store these states.
invented entities (1)
  • TransMem memory module (learned projection W∆^K, transformer block f_m^K, and gating network f_g^K) independent evidence
    purpose: To compute a memory shift ∆h from sparse historical hidden states and apply it to current hidden states via Eq. (7).
    The module is a new parametric component; its existence and design choices are justified by the benchmark improvements, especially cross-domain generalization from HotpotQA to LoCoMo and MemoryAgentBench, which provides evidence beyond the training distribution.

pith-pipeline@v1.3.0-daily-deepseek · 16245 in / 13701 out tokens · 127313 ms · 2026-08-03T14:57:19.181522+00:00 · methodology

0 comments
read the original abstract

Large language model (LLM) agents increasingly operate over long interaction histories, where effective reasoning requires identifying and exploiting task-relevant evidence distributed across past observations and actions. However, useful information encoded in previously computed representations is often underutilized during subsequent generation. We propose \textbf{TransMem}, a lightweight inference-time parametric memory module that transforms sparse historical hidden states from a frozen LLM backbone into reusable memory representations. TransMem uses a lightweight gating network to dynamically apply the latent intervention to the current hidden states, without repeatedly encoding the preceding context. To learn transferable memory utilization rather than task-specific knowledge, we introduce evidence-conditioned self-distillation. A memory-augmented student processes the full context and matches the predictive distribution of an evidence-only teacher that shares the same frozen backbone. Experiments on LoCoMo, HotpotQA, and MemoryAgentBench demonstrate consistent improvements across different model architectures and scales. TransMem yields gains of 11.58--29.25 $F_1$ on LoCoMo and 10.20--13.03 $F_1$ on HotpotQA, while improving the average MemoryAgentBench accuracy from 29.54\% to 40.00\%. These results establish sparse historical hidden states as an effective and efficient memory substrate for long-context LLM agents. Our code is available at https://github.com/Haodong-Lei-Ray/TransMem.

Figures

Figures reproduced from arXiv: 2607.29032 by Botian Shi, Ding Wang, Haodong Lei, Hongsong Wang, Junming Liu, Pinlong Cai, Yirong Chen.

Figure 1
Figure 1. Figure 1: Two properties of contextual representations in LLMs. (a) Hidden states aggregate preceding context. Owing to the autoregressive computation of causal Transformers, each hidden state is computed from its preceding tokens. Consequently, later hidden states can serve as contextual representations of the preceding sequence. (b) Transformer representations are region-sensitive. Excluding the initial sink posit… view at source ↗
Figure 2
Figure 2. Figure 2: The overall architecture of the TransMem framework. (a) Inference with TransMem. TransMem is applied after the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Efficiency comparison in terms of computation [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise gate values for Qwen3-4B Last-4 Trans [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 4 canonical work pages

  1. [1]

    Look Back to Reason Forward: Revisitable Memory for Long-Context

    Yaorui Shi and Yuxin Chen and Siyuan Wang and Sihang Li and Hengxing Cai and Qi GU and Xiang Wang and An Zhang , booktitle=. Look Back to Reason Forward: Revisitable Memory for Long-Context

  2. [2]

    MemAgent: Reshaping Long-Context

    Hongli Yu and Tinghong Chen and Jiangtao Feng and Jiangjie Chen and Weinan Dai and Qiying Yu and Ya-Qin Zhang and Wei-Ying Ma and Jingjing Liu and Mingxuan Wang and Hao Zhou , booktitle=. MemAgent: Reshaping Long-Context

  3. [3]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , title =

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , title =. 2024 , publisher =

    2024

  4. [4]

    Second Conference on Language Modeling , year=

    Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents , author=. Second Conference on Language Modeling , year=

  5. [5]

    2025 , eprint=

    Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution , author=. 2025 , eprint=

    2025

  6. [6]

    International Conference on Learning Representations , year=

    LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory , author=. International Conference on Learning Representations , year=

  7. [7]

    arXiv preprint arXiv:2203.15556 , year=

    Training compute-optimal large language models , author=. arXiv preprint arXiv:2203.15556 , year=

    Pith/arXiv arXiv

  8. [8]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    Pith/arXiv arXiv 2001

  9. [9]

    2025 , eprint=

    Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics , author=. 2025 , eprint=

    2025

  10. [10]

    2025 , eprint=

    From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs , author=. 2025 , eprint=

    2025

  11. [11]

    2025 , issue_date =

    Zhang, Zeyu and Dai, Quanyu and Bo, Xiaohe and Ma, Chen and Li, Rui and Chen, Xu and Zhu, Jieming and Dong, Zhenhua and Wen, Ji-Rong , title =. 2025 , issue_date =. doi:10.1145/3748302 , journal =

    doi:10.1145/3748302 2025

  12. [12]

    2026 , eprint=

    MemCoT: Test-Time Scaling through Memory-Driven Chain-of-Thought , author=. 2026 , eprint=

    2026

  13. [13]

    2026 , eprint=

    Coarse-to-Fine Grounded Memory for LLM Agent Planning , author=. 2026 , eprint=

    2026

  14. [14]

    and Lin, Kevin and Hewitt, John and Paranjape, Ashwin and Bevilacqua, Michele and Petroni, Fabio and Liang, Percy

    Liu, Nelson F. and Lin, Kevin and Hewitt, John and Paranjape, Ashwin and Bevilacqua, Michele and Petroni, Fabio and Liang, Percy. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. 2024

    2024

  15. [15]

    2026 , eprint=

    Memory in the Age of AI Agents , author=. 2026 , eprint=

    2026

  16. [16]

    2024 , eprint=

    MemGPT: Towards LLMs as Operating Systems , author=. 2024 , eprint=

    2024

  17. [17]

    Akari Asai and Zeqiu Wu and Yizhong Wang and Avirup Sil and Hannaneh Hajishirzi , booktitle=. Self-. 2024 , url=

    2024

  18. [18]

    International Conference on Learning Representations , year=

    ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory , author=. International Conference on Learning Representations , year=

  19. [19]

    2025 , eprint=

    MemEvolve: Meta-Evolution of Agent Memory Systems , author=. 2025 , eprint=

    2025

  20. [20]

    2026 , eprint=

    SimpleMem: Efficient Lifelong Memory for LLM Agents , author=. 2026 , eprint=

    2026

  21. [21]

    2026 , eprint=

    MeMo: Memory as a Model , author=. 2026 , eprint=

    2026

  22. [22]

    2025 , eprint=

    MemVerse: Multimodal Memory for Lifelong Learning Agents , author=. 2025 , eprint=

    2025

  23. [23]

    2026 , eprint=

    HyMem: Hybrid Memory Architecture with Dynamic Retrieval Scheduling , author=. 2026 , eprint=

    2026

  24. [24]

    Advances in Neural Information Processing Systems , year =

    Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models , author =. Advances in Neural Information Processing Systems , year =

  25. [25]

    2026 , eprint=

    MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning , author=. 2026 , eprint=

    2026

  26. [26]

    2026 , eprint=

    MEMTS: Internalizing Domain Knowledge via Parameterized Memory for Retrieval-Free Domain Adaptation of Time Series Foundation Models , author=. 2026 , eprint=

    2026

  27. [27]

    2026 , eprint=

    Mem- : Adaptive Memory through Learning When and What to Generate , author=. 2026 , eprint=

    2026

  28. [28]

    2025 , eprint=

    MemGen: Weaving Generative Latent Memory for Self-Evolving Agents , author=. 2025 , eprint=

    2025

  29. [29]

    2026 , eprint=

    Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memory , author=. 2026 , eprint=

    2026

  30. [30]

    Decoupled Reasoning with Implicit Fact Tokens ( DRIFT ): A Dual-Model Framework for Efficient Long-Context Inference

    Xie, Wenxuan and Wang, Xuhong. Decoupled Reasoning with Implicit Fact Tokens ( DRIFT ): A Dual-Model Framework for Efficient Long-Context Inference. Findings of the A ssociation for C omputational L inguistics: ACL. 2026

    2026

  31. [31]

    International Conference on Learning Representations , year =

    MLP Memory: A Retriever-Pretrained Memory for Large Language Models , author =. International Conference on Learning Representations , year =

  32. [32]

    2026 , eprint=

    -mem: Efficient Online Memory for Large Language Models , author=. 2026 , eprint=

    2026

  33. [33]

    2026 , eprint=

    OPSDL: On-Policy Self-Distillation for Long-Context Language Models , author=. 2026 , eprint=

    2026

  34. [34]

    and Salakhutdinov, Ruslan and Manning, Christopher D

    Yang, Zhilin and Qi, Peng and Zhang, Saizheng and Bengio, Yoshua and Cohen, William W. and Salakhutdinov, Ruslan and Manning, Christopher D. , booktitle=

  35. [35]

    Evaluating Very Long-Term Conversational Memory of LLM Agents

    Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fang, Yuwei. Evaluating Very Long-Term Conversational Memory of LLM Agents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Long Papers). 2024. doi:10.18653/v1/2024.acl-long.747

    doi:10.18653/v1/2024.acl-long.747 2024

  36. [36]

    Evaluating Memory in

    Yuanzhe Hu and Yu Wang and Julian McAuley , booktitle=. Evaluating Memory in

  37. [37]

    2025 , eprint=

    Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory , author=. 2025 , eprint=

    2025

  38. [38]

    Vicky Zhao and Lili Qiu and Dongmei Zhang

    Zhuoshi Pan and Qianhui Wu and Huiqiang Jiang and Menglin Xia and Xufang Luo and Jue Zhang and Qingwei Lin and Victor Ruhle and Yuqing Yang and Chin-Yew Lin and H. Vicky Zhao and Lili Qiu and Dongmei Zhang. LLML ingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. Findings of the Association for Computational Linguistics...

    2024

  39. [39]

    Advances in neural information processing systems , volume=

    Retrieval-augmented generation for knowledge-intensive nlp tasks , author=. Advances in neural information processing systems , volume=

  40. [40]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Memorybank: Enhancing large language models with long-term memory , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  41. [41]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. International Conference on Learning Representations , volume=

  42. [42]

    2026 , eprint=

    Understanding LoRA as Knowledge Memory: An Empirical Analysis , author=. 2026 , eprint=

    2026

  43. [43]

    and Schuetze, Hinrich and Tresp, Volker and Ma, Yunpu

    Yan, Sikuan and Yang, Xiufeng and Huang, Zuchao and Nie, Ercong and Ding, Zifeng and Li, Zonggen and Ma, Xiaowen and Bi, Jinhe and Kersting, Kristian and Pan, Jeff Z. and Schuetze, Hinrich and Tresp, Volker and Ma, Yunpu. Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning. Annual Meeting of the A sso...

    work page doi:10.18653/v1/2026.acl-long.583 2026

  44. [44]

    arXiv preprint arXiv:2605.19952 , year =

    Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory , author =. arXiv preprint arXiv:2605.19952 , year =

    Pith/arXiv arXiv

  45. [45]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  46. [46]

    Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents

    Yu, Yi and Yao, Liuyi and Xie, Yuexiang and Tan, Qingquan and Feng, Jiaqi and Li, Yaliang and Wu, Libing. Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents. Annual Meeting of the A ssociation for C omputational L inguistics (Long Papers). 2026. doi:10.18653/v1/2026.acl-long.981

    work page doi:10.18653/v1/2026.acl-long.981 2026

  47. [47]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  48. [48]

    2025 , eprint=

    Qwen2.5 Technical Report , author=. 2025 , eprint=

    2025

  49. [49]

    Ouyang, Long and Wu, Jeff and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll L. and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and Schulman, John and Hilton, Jacob and Kelton, Fraser and Miller, Luke and Simens, Maddie and Askell, Amanda and Welinder, Peter and Christiano, Paul and Leike, Jan and Lowe, R...

    2022

  50. [50]

    2024 , eprint=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

    2024

  51. [51]

    2026 , eprint=

    Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe , author=. 2026 , eprint=

    2026

  52. [52]

    International Conference on Machine Learning , year =

    Yuhui Li and Fangyun Wei and Chao Zhang and Hongyang Zhang , title =. International Conference on Machine Learning , year =

  53. [53]

    2025 , eprint=

    DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models , author=. 2025 , eprint=

    2025

  54. [54]

    2025 , eprint=

    DeepSeek-V3 Technical Report , author=. 2025 , eprint=

    2025

  55. [55]

    Found in the middle: Calibrating Positional Attention Bias Improves Long Context Utilization

    Hsieh, Cheng-Yu and Chuang, Yung-Sung and Li, Chun-Liang and Wang, Zifeng and Le, Long and Kumar, Abhishek and Glass, James and Ratner, Alexander and Lee, Chen-Yu and Krishna, Ranjay and Pfister, Tomas. Found in the middle: Calibrating Positional Attention Bias Improves Long Context Utilization. Findings of the Association for Computational Linguistics: A...

    doi:10.18653/v1/2024.findings-acl.890 2024

  56. [56]

    2024 , editor =

    Ding, Yiran and Zhang, Li Lyna and Zhang, Chengruidong and Xu, Yuanyuan and Shang, Ning and Xu, Jiahang and Yang, Fan and Yang, Mao , booktitle =. 2024 , editor =

    2024

  57. [57]

    2025 , eprint=

    Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects , author=. 2025 , eprint=

    2025

  58. [58]

    T hink SLM : Towards Reasoning in Small Language Models

    Srivastava, Gaurav and Cao, Shuxiang and Wang, Xuan. T hink SLM : Towards Reasoning in Small Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.1659

    work page doi:10.18653/v1/2025.emnlp-main.1659 2025

  59. [59]

    2026 , eprint=

    effGen: Enabling Small Language Models as Capable Autonomous Agents , author=. 2026 , eprint=

    2026

  60. [60]

    HELENE : Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization

    Zhao, Huaqin and Li, Jiaxi and Pan, Yi and Liang, Shizhe and Yang, Xiaofeng and Dou, Fei and Liu, Tianming and Lu, Jin. HELENE : Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/20...

    work page doi:10.18653/v1/2025.emnlp-main.1323 2025