REVIEW 2 major objections 2 minor 24 references
Learning What Not to Forget: Long-Horizon Agent Memory from a Few Kilobytes of Learning
T0 review · 2 major / 2 minor · reviewed 2026-06-26 · grok-4.3
Pith's one-line read A few-kilobyte CPU-only scorer learns to retain only load-bearing history units and matches full-history accuracy for long-horizon LLM agents.
desk verdict LRE shows a lightweight learned eviction policy can handle agent memory effectively in the tested scenarios. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
LRE (Learned Relevance Eviction), a small CPU-only scorer that ranks history units for verbatim retention without access to future queries.
What would settle it
A controlled run on a fresh task in which the LRE policy drops a detail required for success while the full-history policy retains it, producing a measurable drop in completion rate.
Extended reading notes
Core claim
LRE is a language-model-free scorer, a few kilobytes in size, that is trained to identify which units of accumulated interaction history are load-bearing and then retains those units by verbatim extraction. In agent experiments it matches the task accuracy obtained by keeping the entire history, exceeds that no-eviction baseline by 27 percent on the simplest tasks, finishes one looping task in 37 percent fewer calls, and solves 14 tasks that no other policy solves; on conversational memory it outperforms dense and token-pruning encoders at zero neural cost; and on LoCoMo reading it delivers the best budgeted answer quality while using 68 percent fewer tokens. Training can be annotation-free:
Load-bearing premise
A scorer trained only on the system’s own past behavior can reliably identify which history units will be load-bearing for unseen future tasks.
Editorial extensions
If this is right
- On agent tasks LRE matches the accuracy of retaining the entire history while cutting peak context size up to 52 percent.
- On the simplest tasks it exceeds the no-eviction baseline accuracy by 27 percent with zero compressor calls.
- It finishes tasks where other policies loop and solves 14 tasks that no other run policy solves.
- Annotation-free training on the agent’s own traces recovers 95 percent of the supervised scorer’s performance.
- On conversational memory LRE outranks dense and token-pruning encoders at zero neural cost.
Reading between the lines
- The same lightweight ranking approach could be applied to other sequential systems that must discard state without knowing the next query in advance.
- Because eviction decisions are made from past traces alone, the method may scale to multi-agent or distributed settings where sharing full histories is costly.
- If the scorer generalizes across domains, it could reduce the need for ever-larger context windows in production agents.
- The verbatim-retention design preserves exact state, which may matter for tasks where even small paraphrases break downstream correctness.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces LRE (Learned Relevance Eviction), a few-kilobyte, CPU-only, language-model-free scorer that learns to identify and retain load-bearing units of interaction history for long-horizon LLM agents via verbatim extraction. It claims that under matched-budget comparisons, LRE matches the accuracy of retaining full history overall, exceeds the no-eviction baseline by 27% on simplest tasks, reduces peak context size by up to 52% with zero compressor calls, completes tasks where baselines loop (e.g., 37% fewer calls on one trace, solving 14 tasks no other policy solves), outperforms dense/token-pruning encoders on conversational memory at zero neural cost, achieves best budgeted answer quality on LoCoMo while using 68% fewer tokens, and recovers 95% of supervised effectiveness under annotation-free training on the system's own behavior.
Significance. If the reported empirical results hold under the described conditions, the work offers a practical, low-overhead solution to the fidelity problem of memory eviction in LLM agents, where future queries are unavailable and exact state retention is decisive. The annotation-free training result and outperformance on agent traces and LoCoMo are notable strengths, as is the explicit positioning against compressor-based and full-history baselines.
major comments (2)
- [Abstract and experimental results] The central claim that the scorer generalizes to unseen future tasks without access to the future query itself (weakest assumption noted in the reader's report) requires explicit validation in the experimental section; the abstract reports 95% recovery under annotation-free training, but without details on how training traces were constructed or held-out task distributions, it is unclear whether this tests the no-future-query regime or merely replays similar behaviors.
- [Abstract] Soundness is limited by the absence of methodological details, error bars, dataset descriptions, and statistical significance tests in the reported performance numbers (e.g., the 27% improvement and 52% context reduction); these must be supplied with precise definitions of 'simplest tasks,' agent traces, and matched-budget controls to allow evaluation of the accuracy-cost plane claims.
minor comments (2)
- Clarify the exact architecture and training objective of the 'few kilobytes' scorer, including input features and how 'load-bearing' labels are derived in the annotation-free case.
- Provide the full set of baselines and their implementation details for the conversational memory and LoCoMo evaluations to ensure reproducibility.
Simulated Author's Rebuttal
We thank the referee for the constructive review. We address each major comment below with targeted revisions to improve clarity and completeness while preserving the manuscript's core contributions.
read point-by-point responses
-
Referee: [Abstract and experimental results] The central claim that the scorer generalizes to unseen future tasks without access to the future query itself (weakest assumption noted in the reader's report) requires explicit validation in the experimental section; the abstract reports 95% recovery under annotation-free training, but without details on how training traces were constructed or held-out task distributions, it is unclear whether this tests the no-future-query regime or merely replays similar behaviors.
Authors: The annotation-free regime trains exclusively on the agent's own past interaction traces, which by design contain no information about future queries. Evaluation occurs on held-out task distributions drawn from the same benchmark but excluded from training. We will expand the experimental section with a dedicated subsection detailing trace construction (sequences of agent actions and observations), the train/test split (chronological or task-ID based to ensure no leakage), and explicit confirmation that scoring and training use only historical data. This will directly validate the no-future-query generalization claim. revision: yes
-
Referee: [Abstract] Soundness is limited by the absence of methodological details, error bars, dataset descriptions, and statistical significance tests in the reported performance numbers (e.g., the 27% improvement and 52% context reduction); these must be supplied with precise definitions of 'simplest tasks,' agent traces, and matched-budget controls to allow evaluation of the accuracy-cost plane claims.
Authors: The body of the manuscript already defines 'simplest tasks' (shortest-horizon agent traces), 'agent traces' (full sequences of LLM calls, observations, and memory states), and matched-budget controls (identical token limits across policies). However, the abstract omits error bars, significance tests, and concise definitions. We will revise the abstract to incorporate brief definitions and add error bars plus statistical tests (e.g., paired t-tests) to the key quantitative claims in the experimental section and a new summary table. Some methodological depth will remain in the main text due to abstract length limits, but the abstract will become self-contained for the reported numbers. revision: partial
Circularity Check
No circularity; empirical method with external validation
full rationale
The paper describes an empirical learned scorer (LRE) for history eviction, trained on agent traces or self-behavior and evaluated on accuracy, cost, and task completion metrics across agent and conversational benchmarks. No equations, derivations, or uniqueness theorems are presented that could reduce to self-definition or fitted inputs. All load-bearing claims rest on reported experimental outcomes (e.g., 95% recovery under annotation-free training, 27% outperformance on simple tasks) rather than any internal construction that equates prediction to input by design. Self-citations are absent from the provided text, and the method is externally falsifiable via the described benchmarks.
Assumptions & free parameters
Cite this review
Pith. "Pith review of Learning What Not to Forget: Long-Horizon Agent Memory from a Few Kilobytes of Learning." pith.science (2026). https://pith.science/paper/WZXV42ZW
@misc{pith2026260620954,
author = {Pith},
title = {Pith review of: Learning What Not to Forget: Long-Horizon Agent Memory from a Few Kilobytes of Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/WZXV42ZW}},
note = {Machine review of arXiv:2606.20954}
}
read the original abstract
Long-running language-model systems accumulate interaction history that outgrows the context window, so they must continually evict. When an eviction policy drops a load-bearing detail, for example an access token issued at login or a path the next call needs, the action fails. We present LRE (Learned Relevance Eviction), a few kilobytes, CPU-only, language-model-free scorer that learns which units of history are load-bearing and keeps them by verbatim extraction. Under a matched-budget comparison, in our experiment, no baseline dominates LRE on the accuracy-cost plane. On agents, LRE matches the accuracy of keeping the entire history overall. On the simplest tasks, it exceeds that no-eviction baseline by 27%, while requiring zero compressor calls and reducing peak context size by up to 52%. A controlled study trace shows LRE completes tasks where the others loop, finishing one such task in 37% fewer calls than keeping everything and solving 14 tasks where no other run policy does. On conversational memory, LRE outranks dense and token-pruning encoders at zero neural cost. In downstream evaluation, LRE gives the best budgeted answer quality on LoCoMo reading 68% fewer tokens. Its supervision can also be annotation-free: training only on the system's own behavior recovers 95% of the supervised scorer's effectiveness. We argue that, because memory eviction in LLM agents is a fidelity problem, it requires a deployable proactive policy where the future query is unavailable and exact state is decisive, and that cheap learned relevance can be sufficient.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
What Deserves Memory: Adaptive Memory Distillation for LLM Agents
Evaluating very long-term conversational memory of llm agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851– 13870. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochasti- cally larger than the other.The annals of mathemati- ca...
work page Pith review arXiv 1947
-
[2]
In Findings of the Association for Computational Lin- guistics: ACL 2024, pages 963–981
Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Findings of the Association for Computational Lin- guistics: ACL 2024, pages 963–981. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bern- stein. 2023. Generative agents: Interactive simulacra of human behavi...
2024
-
[3]
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Appworld: A controllable world of apps and people for benchmarking interactive coding agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16022–16076. Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. 2024. Longmemeval: Benchmarking chat assistants on long-...
work page Pith review arXiv 2024
-
[4]
Agentfold: Long-horizon web agents with proactive context management.arXiv preprint arXiv:2510.24699. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuan- dong Tian, Christopher Ré, Clark Barrett, and 1 oth- ers. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Ad- vance...
-
[5]
Data structure.Each sample corresponds to a dialogue turn ui, and all turns within a conversation form a group g∈ {1, . . . ,10} . The task is to assign a relevance score to each turn without conditioning on a specific downstream query. About 24% of dialogue turns are considered truly relevant (gold evidence across questions)
-
[6]
In the supervised setting, a turn is labeled positive (yi = 1) if it appears in the gold evidence set of any question associated with its conversation
Label construction.We consider two labeling regimes. In the supervised setting, a turn is labeled positive (yi = 1) if it appears in the gold evidence set of any question associated with its conversation. Because these labels are pooled across all the questions tied to the conversation, it yields a query-agnostic relevance signal. The model learns to iden...
-
[7]
Bye!”, “Take care!
Splitting and leakage-free training.We use a Leave-One-Group-Out (LOGO) protocol over the 10 conversations. In each fold, one full conversation is held out for evaluation while the remaining nine are used for training. To prevent information leakage, the vectorizer and StandardScaler are fitted strictly on training data within each fold. A deduplication g...
-
[8]
A logistic regression model is trained to estimate relevance scores pi =P(y i = 1|u i) using these features
Feature representation and scoring.Each unit is mapped to a feature vector combining: (i) a TF-IDF representation of the text, and (ii) six trajectory features: position, recency, length, digit count, question-marker presence, and capitalized-word count (Table 6). A logistic regression model is trained to estimate relevance scores pi =P(y i = 1|u i) using...
Show all 24 references
-
[9]
For intrinsic evaluation, we compute ROC-AUC over held-out folds (Section E)
Evaluation and downstream usage.The model is evaluated both intrinsically and extrinsically. For intrinsic evaluation, we compute ROC-AUC over held-out folds (Section E). For extrinsic evaluation, retained dialogue turns are passed verbatim into a downstream QA model. Performa...
2024
-
[10]
Each group consists of approximately 50 sessions forming a retrieval haystack
Data structure and eviction units.The dataset (longmemeval_s) contains up to 500 questions, where each question defines an independent group. Each group consists of approximately 50 sessions forming a retrieval haystack. The task is highly sparse, with only 5% of sessions are ...
-
[11]
In the supervised setting, a session is labeled positive (yi = 1) if its session ID appears in the question- specific answer_session_ids
Label construction. In the supervised setting, a session is labeled positive (yi = 1) if its session ID appears in the question- specific answer_session_ids. In the self-supervised setting, a proxy label is assigned if the session contains at least 40% token overlap with the e...
-
[12]
high computational cost of full leave-one-group-out evaluation over 500 questions
Splitting and leakage prevention.We use grouped K-fold cross-validation with k= 5 due to the 11 Require:LongMemEval S datasetD={(u i, yi, gi)}with up to 500 questions Require:TF-IDF vectorizerϕ, scalerσ, classifierf Require:Dedup flagdedup 1:Dataset structure 2:Each groupg i i...
-
[13]
A logistic regression model outputs relevance scores pi =P(y i = 1|u i) without access to the downstream question context
Feature representation and scoring.Each session is represented using a fusion of: (i) TF-IDF lexical features, and (ii) six standardized trajectory features. A logistic regression model outputs relevance scores pi =P(y i = 1|u i) without access to the downstream question context
-
[14]
For downstream evaluation, retained sessions are passed to a QA model (gpt-4.1-mini), and responses are scored using a stronger LLM judge (gpt-4o)
Evaluation protocol.LongMemEval S is evaluated on both retrieval quality and downstream task performance.For relevance prediction, we compute ROC-AUC. For downstream evaluation, retained sessions are passed to a QA model (gpt-4.1-mini), and responses are scored using a stronge...
2025
-
[15]
This produces full execution trajectories containing interleaved Python code actions and environment observations, which are used as training data for the LRE scorer
Trajectory generation and training data.We first execute an unevicted baseline agent ("nocomp") on the AppWorld training split consisting of 90 tasks. This produces full execution trajectories containing interleaved Python code actions and environment observations, which are u...
-
[16]
Let zi denote an identifier introduced at stepi
Self-supervised labeling via quotation reuse.Since no human relevance annotations are available, we construct a zero-supervision label using a hindsight reuse criterion. Let zi denote an identifier introduced at stepi. The label is defined as: •y i = 1ifz i is reused in at lea...
-
[17]
Scorer training.The logistic regression scorer ( s90) is evaluated under a Leave-One-Task-Out (LOGO) protocol
-
[18]
When the context length exceeds the token budget (B= 2048 ), the eviction module is triggered
Inference and eviction loop.During evaluation on test task splits, the agent operates in a ReAct-style loop. When the context length exceeds the token budget (B= 2048 ), the eviction module is triggered. LRE performs CPU-based scoring of historical units. Units are ranked by s...
-
[19]
keep everything
Evaluation and Task Goal Completion (TGC).AppWorld evaluates success using Task Goal Completion (TGC), computed via hidden unit tests that operate on the final environment state. A known failure mode isself-reported success bias, where task success is determined from the agent...
-
[20]
Keep all sections relevant and concise
-
[21]
Use reusable structured formats when summarizing artifacts
-
[22]
Ensure agent can resume task with no loss of information
-
[23]
Include key info from errors or failed attempts to prevent repeated mistakes
-
[24]
Request $28 privately on Venmo from my roommate, Melissa, with a note, ‘For the movie tickets’
Preserve all essential artifacts and data needed to complete the task. ### [Output Format] Do not include the input or any additional explanation. Only return the formatted summary. C.2 Conversational domain Every conversational method reduces to a per-unitkeep-score(higher me...
2024
Reviewed June 26, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.