Machine Learning · arXiv:2611.01747 v1 · submitted 1 Nov 2026 · 104 references
Evolving Memory, Learning to Meta-Control: Test-Time Adaptation for Experience-augmented LLM Agents
The paper says
“Self-evolving agents accumulate experience from task streams, but existing frameworks typically rely on fixed pipelines to retrieve and utilize this experience, neglecting when and how to deploy it.”
more of the abstract
We introduce Online Meta-Control of Experience (OMCE), a framework that updates experience memory and learns a meta-control policy from task feedback during test-time adaptation. OMCE employs a lightweight contextual-bandit meta-controller that uses compact retrieval signals to select among four cognitive actions: DIRECT, EPISODE, GUIDE, and DISTILL. This routing mechanism operationalizes three coupled decisions: whether to invoke experience, which memory format to utilize, and when to consolidate new guidance. OMCE evolves online, leveraging task feedback to update the memory bank and refine the routing policy via contextual bandit updates. Experiments on four online adaptation benchmarks across four backbones show that OMCE improves first-attempt success by up to 20.5 percentage points over the strongest compared baseline, while reducing per-task token use by a factor of 5.7 to 62.5 relative to ACE with GPT-5.2. Controlled ablations further support the benefits of learned routing and continued routing policy adaptation.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 4 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Solid paper: clean formulation of a real problem (learned routing over experience), backed by extensive experiments and a theoretical bound. The meta-state assumption is a soft spot but not a fatal one. This line is wrong
2 major objections · 3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 104 references say.
The paper says, at §3.2, Eq. (1) and (9); meta-state definition
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The meta-state is a hand-crafted 9-dimensional vector that condenses retrieval scores, quality scores, and outcome diversity. It never reads the actual text of retrieved memories. The paper does not ablate this representation (e.g., by comparing to a version that includes text embeddings or a learned representation).
more of objection 1
While the empirical results are strong and suggest that the summary statistics suffice for the tasks tested, the claim that the controller learns 'when and how to use experience' implicitly assumes that all task-relevant variation for routing is captured by these nine numbers. A direct ablation or a discussion of why text content is unnecessary (or why the summary is sufficient) would strengthen the paper. I consider this a missing experiment, but not a fatal flaw given the overall empirical support.
The paper says, at §4.1 and Appendix G.4
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.91):
“The 100 selected BigCodeBench tasks remain part of the final evaluation stream, so the BigCodeBench result contains a limited overlap between hyperpa- rameter selection and evaluation.”
Pith says: Objection 2 of 2 · major
The hyperparameter configuration (Table 5) was selected on a 100-task subset of BigCodeBench, and those same 100 tasks remain in the final evaluation stream. The authors acknowledge this overlap (Appendix G.4: 'the BigCodeBench result contains a limited overlap between hyperparameter selection and evaluation').
more of objection 2
For a single configuration shared across all benchmarks and backbones, this is acceptable, but the paper should quantify the potential for optimistic bias on BigCodeBench, e.g., by reporting the performance on the held-out portion separately. A brief sentence clarifying how much of the 1,140-task stream is the 'selection' subset would suffice.
3 minor objections
The paper says, at §4.3, Table 3
Pith says: minor objection 1 of 3
The controlled analysis (shared memory snapshot) uses only GPT-5.2. The authors note this limitation in the main text (line: 'Our controlled comparisons of memory use and controller updates focus on GPT-5.2, while the routing comparison spans all four backbones'). Adding the same analysis on at least one more backbone (e.g., DeepSeek-V3.2) would strengthen the generality claim. As is, this is a minor completeness concern.
The paper says, at Appendix E
Pith says: minor objection 2 of 3
The theoretical regret bound (Theorem E.2) assumes the stationary regime (γ=1) and analyzes an 'optimistic instantiation' (LinUCB-style) rather than the Thompson Sampling used in experiments. While the connection is plausible, the paper should state explicitly that the bound does not directly cover the experimental setting (non-stationary with TS) and that it serves as an intuition for why the linear bandit can learn. The current Remark acknowledges this in part but could be clearer.
show 1 more
The paper says, at Figure 3 and Figure 5
Pith says: minor objection 3 of 3
The x-axis label in the performance–efficiency plots is 'One-shot Success Rate (OSR)' but the markers include both OSR and CSR for some methods. Please ensure the legend and axis labels match the reported metrics. For example, Reflexion is plotted only on CSR, which is fine, but the figure caption should clarify what is shown.
What would settle it
Augment the meta-state with the mean embedding of the top-M retrieved entries (e.g., from the same text-embedding-3-large used for retrieval), adding two extra dimensions (episodic mean and guidance mean). Compare OMCE-TS with this augmented state against the original 9-d version on all four benchmarks with GPT-5.2 (one run each, as in the paper). If the augmented version does not improve OSR by more than 1 percentage point on any benchmark, the 9-d state is sufficient. If it does improve, the paper's representation is a limiting factor and the central claim should be qualified.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 4 Nov 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 2 major objections, 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is a solid, well-executed paper. The core idea—learning when and how to use accumulated experience via a lightweight contextual bandit—is genuinely new and practically motivated. The authors formalize three coupled decisions (invocation, form, consolidation) that prior work either hardwires or ignores. The experiments are thorough: four benchmarks, four backbones, multiple baselines, and controlled ablations that isolate the routing policy from memory content. The token-efficiency gains are striking. The theoretical regret bound, while for a stationary simplification, adds rigor.
The main soft spot is the meta-state representation. The 9-dimensional vector condenses retrieval scores, quality scores, and outcome diversity, but never reads the actual memory text. The authors do not test whether a richer representation (e.g., including embeddings of retrieved content) would further improve performance. This is the stress-test concern, and it is legitimate: the meta-state could conflate semantically different memories with similar scores. However, the empirical results are strong enough that this assumption, while untested, has not obviously bottlenecked performance. The paper would be stronger with an ablation that adds text embeddings, but the absence is not a dealbreaker—the current representation already works well across diverse tasks.
Other minor issues: hyperparameter selection overlaps with BigCodeBench (acknowledged), no error bars (a real limitation but common in LLM API experiments), and the theoretical analysis assumes a stationary regime (authors note this). These are proportionately small concerns.
This paper is for researchers working on self-evolving agents, memory-augmented LLMs, and meta-cognitive control. It deserves a serious referee. The central claim holds up, the evidence is reproducible (code presumably released), and the framing is likely to influence subsequent work.
The strongest objection
The central claim that OMCE learns when and how to use experience depends critically on the meta-state representation (Section 3.2, Equation 1 and 9). The meta-state is a hand-crafted 9-dimensional vector that condenses retrieval scores, outcome-based quality scores, and outcome-type diversity. Crucially, it never reads the actual text of retrieved episodic records or guidance entries. The linear bandit therefore bases its decision on summary statistics that may conflate semantically different memories that happen to have similar scores. For example, two episodic records with the same retrieval score and outcome type could contain very different lessons; the meta-state treats them identically. The paper does not provide an ablation that validates this compression (e.g., by comparing to a version that includes retrieved-text embeddings or a learned representation). While the empirical results are strong, the assumption that all task-relevant variation is captured by these nine numbers remains unexamined. If the meta-state is insufficient, the bandit's learning is bottlenecked, and the observed gains might be further improvable with a richer representation, or might degrade on tasks where score-alone signals are misleading.
The reader correctly identified the sufficiency of the meta-state as the weakest assumption. My concern sharpens this: the meta-state discards semantic content entirely, which is a design choice that could be a bottleneck for learning the optimal routing policy. However, the paper provides extensive empirical evidence that the learned routing already outperforms fixed policies and a hand-crafted heuristic across four benchmarks and four backbones. The absence of a direct validation of the representation is a gap, but the experimental results partly address the concern by showing that the representation works well in practice. Therefore, the reader's ACCEPT verdict is still justified; the concern suggests a natural future direction rather than a fundamental flaw. I concur with the reader that the paper is sound and the central claim is supported. The concrete test would strengthen the paper by ruling out a plausible alternative explanation, but its absence does not invalidate the current results.
The referee's report
If the claimed results hold, this work makes a meaningful contribution by reframing experience-augmented reasoning as a learnable meta-control problem. The empirical evidence is extensive: four diverse benchmarks, four backbones, multiple baselines (including fixed policies, heuristics, and shared-memory controls), and controlled ablations separate the contribution of routing from memory content. The theoretical regret bound (Appendix E, for a stationary simplified version) adds rigor. The performance–token trade-off is particularly compelling, showing order-of-magnitude efficiency gains relative to ACE while maintaining or improving accuracy. The paper identifies a genuine bottleneck in current self-evolving agents and proposes a lightweight, practical solution.
Pith's reading of the paper (a machine's summary, not the authors')
Self-evolving LLM agents accumulate experience from task streams, but existing frameworks retrieve and inject that experience unconditionally, wasting tokens and sometimes harming performance. This paper introduces Online Meta-Control of Experience (OMCE), a framework that treats the decision of whether and how to use experience as a learnable meta-control problem. OMCE uses a lightweight contextual bandit that reads a nine-dimensional summary of retrieval quality and outcome diversity to choose among four actions: direct execution (no memory), episodic reuse, guidance reuse, or distillation of new guidance. The bandit policy is updated online from task reward signals, allowing it to adapt as memory evolves. Experiments across four benchmarks and four backbone LLMs show that OMCE improves first-attempt success by up to 20.5 percentage points over the strongest baseline and reduces per-task token consumption by a factor of 5.7 to 62.5 relative to the playbook-based ACE method.
What carries the argument. The key machinery is a contextual-bandit meta-controller with linear reward models. The meta-state is a nine-dimensional vector composed of: mean retrieval scores from episodic and guidance memory (relevance), mean quality scores and their relevance-weighted counterparts (quality), and the balance and normalized entropy of outcome types among retrieved episodic records (outcome diversity). The controller uses either Thompson Sampling or LinUCB to select among four cognitive actions. The bandit statistics are updated online with a forgetting factor after each task, using the observed reward. The entire meta-controller operates on small matrices and requires no additional LLM calls beyond the standard task execution and memory evolution.
Core claim. The paper argues that the bottleneck in self-evolving agents is not accumulating richer experience but learning when and how to deploy it. The central claim is that a contextual-bandit meta-controller operating on compact meta-state features—relevance, quality, and outcome diversity of retrieved memory—can learn to select among DIRECT, EPISODE, GUIDE, and DISTILL actions, and that this learned routing improves first-attempt success across diverse benchmarks and backbone models while drastically reducing token cost compared to always-inject baselines. The authors demonstrate that no fixed experience-use strategy is consistently best, and that the learned controller outperforms both fixed policies and a hand-crafted heuristic rule. Ablations confirm that the gains stem from the routing policy, not merely the memory content, and that continued policy adaptation as memory evolves is necessary for best performance.
The premise it rests on. The paper assumes that a nine-dimensional vector of retrieval scores, quality scores, and outcome diversity captures all information needed to decide whether and how to use retrieved experience, without the controller ever reading the actual content of the memories.
If the paper is right (Pith's inferences, not the authors')
Self-evolving agents can be designed with a lightweight meta-controller that learns to gate experience, improving efficiency and robustness across different task types.
The meta-control approach suggests that the value of experience is not uniform; agents should dynamically choose between raw cases, distilled principles, and direct execution rather than always injecting memory.
The substantial token savings (5.7–62.5×) imply that learned routing can make test-time adaptation practical for cost-sensitive deployments where API call budgets are limited.
The finding that indiscriminate retrieval can hurt performance (worse than no memory) underscores the need for selective invocation, and that routing policy must be learned, not hard-coded.
The success of a linear model on nine hand-crafted features suggests that effective meta-control does not require full access to memory content, opening the door to even simpler controllers.
The nine-dimensional meta-state may encode sufficient structure for the four benchmarks tested, but could miss information (e.g., semantic novelty or task difficulty) that matters on other domains; testing on a broader distribution is needed.
The paper assumes linear payoff functions; richer nonlinear models or learned embeddings might improve performance on streams with complex dependencies, though at the cost of interpretability and compute.
The forgetting factor (γ=0.98) provides a simple mechanism for non-stationarity; a formal analysis of catastrophic forgetting in the meta-controller under abrupt distribution shifts could reveal failure modes.
The four action set is hand-designed; future work could extend it to actions like 'ignore irrelevant memory' or 'retrieve additional evidence before deciding', potentially yielding further gains.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Self-evolving agents accumulate experience from task streams, but existing frameworks typically rely on fixed pipelines to retrieve and utilize this experience, neglecting when and how to deploy it. We introduce Online Meta-Control of Experience (OMCE), a framework that updates experience memory and learns a meta-control policy from task feedback during test-time adaptation. OMCE employs a lightweight contextual-bandit meta-controller that uses compact retrieval signals to select among four cognitive actions: DIRECT, EPISODE, GUIDE, and DISTILL. This routing mechanism operationalizes three coupled decisions: whether to invoke experience, which memory format to utilize, and when to consolidate new guidance. OMCE evolves online, leveraging task feedback to update the memory bank and refine the routing policy via contextual bandit updates. Experiments on four online adaptation benchmarks across four backbones show that OMCE improves first-attempt success by up to 20.5 percentage points over the strongest compared baseline, while reducing per-task token use by a factor of 5.7 to 62.5 relative to ACE with GPT-5.2. Controlled ablations further support the benefits of learned routing and continued routing policy adaptation.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-03
The authors' main claim, as the classifier picked it out 1.00
“Experiments on four online adaptation benchmarks across four backbones show that OMCE improves first-attempt success by up to 20.5 percentage points over the strongest compared baseline, while reducing per-task token use by a factor of 5.7 to 62.5 relative to ACE with GPT-5.2.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.08 · human participants 0.04 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Self-evolving agents accumulate experience from task streams, but existing frameworks typically rely on fixed pipelines to retrieve and utilize this experience, neglecting when and how to deploy it. | gap 0.78 | 0.06 | 0.03 | asserted 1.00 |
| We introduce Online Meta-Control of Experience (OMCE), a framework that updates experience memory and learns a meta-control policy from task feedback during test-time adaptation. | contribution 0.99 | 0.93 | 0.14 | asserted 1.00 |
| OMCE employs a lightweight contextual-bandit meta-controller that uses compact retrieval signals to select among four cognitive actions: DIRECT, EPISODE, GUIDE, and DISTILL. | method 0.99 | 0.87 | 0.04 | asserted 1.00 |
| This routing mechanism operationalizes three coupled decisions: whether to invoke experience, which memory format to utilize, and when to consolidate new guidance. | method 0.99 | 0.77 | 0.04 | asserted 1.00 |
| OMCE evolves online, leveraging task feedback to update the memory bank and refine the routing policy via contextual bandit updates. | method 0.96 | 0.84 | 0.08 | asserted 1.00 |
| Experiments on four online adaptation benchmarks across four backbones show that OMCE improves first-attempt success by up to 20.5 percentage points over the strongest compared baseline, while reducing per-task token use by a factor of 5.7 to 62.5 relative to ACE with GPT-5.2. | result 1.00 | 0.96 | 0.76 | asserted 1.00 |
| Controlled ablations further support the benefits of learned routing and continued routing policy adaptation. | result 0.85 | 0.93 | 0.09 | asserted 0.93 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-03: 1 request, 5,681 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
References (104)
- [1] Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards on Pith
- [2] Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume = on Pith
- [3] Deep learning , author=. 2016 , publisher= on Pith
- [4] Reflexion: Language agents with verbal reinforcement learning , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [7] Expel: Llm agents are experiential learners , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume= on Pith
- [9] React: Synergizing reasoning and acting in language models , author=. The eleventh international conference on learning representations , year= on Pith
- [14] Dual-process theories of reasoning: Contemporary issues and developmental applications , author=. Developmental review , volume=. 2011 , publisher= on Pith
- [15] Dual-process theories of higher cognition: Advancing the debate , author=. Perspectives on psychological science , volume=. 2013 , publisher= on Pith
- [18] Language models are capable of metacognitive monitoring and control of their internal activations , author=. ArXiv , pages= on Pith
- [20] Agent workflow memory , author=. arXiv preprint arXiv:2409.07429 , year= on PithPith's reviewarXiv
- [22] Retrieval-augmented generation for knowledge-intensive nlp tasks , author=. Advances in neural information processing systems , volume= on Pith
- [23] Distinguishing the reflective, algorithmic, and autonomous minds: Is it time for a tri-process theory , author=. In two minds: Dual processes and beyond , pages= on Pith
- [24] Robertson, Stephen E and Walker, Steve and Jones, Susan and Hancock-Beaulieu, Micheline M and Gatford, Mike and others , journal= on Pith
- [25] Improving language models by retrieving from trillions of tokens , author=. International conference on machine learning , pages=. 2022 , organization= on Pith
- [26] MemEvolve: Meta-Evolution of Agent Memory Systems , author=. arXiv preprint arXiv:2512.18746 , year= on PithPith's reviewarXiv
- [30] Humanity's last exam , author=. arXiv preprint arXiv:2501.14249 , year= on PithPith's reviewarXiv
- [32] LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners , author=. arXiv preprint arXiv:2505.11942 , year= on PithPith's reviewarXiv
- [37] Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages= on Pith
- [38] Big-bench extra hard , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
- [40] ^2 -Bench: Evaluating Conversational Agents in a Dual-Control Environment , author=. 2025 , eprint= on Pith
show 80 more
- [41] Improved algorithms for linear stochastic bandits , author=. Advances in neural information processing systems , volume= on Pith
- [42] Contextual bandits with linear payoff functions , author=. Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages=. 2011 , organization= on Pith
- [43] Thompson sampling for contextual bandits with linear payoffs , author=. International conference on machine learning , pages=. 2013 , organization= on Pith
- [44] Weighted linear bandits for non-stationary environments , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [48] EvoFSM: Controllable Self-Evolution for Deep Research with Finite State Machines , author=. arXiv preprint arXiv:2601.09465 , year= on PitharXiv
- [51] Doing more with less: meta-reasoning and meta-learning in humans and machines , author=. Current Opinion in Behavioral Sciences , volume=. 2019 , publisher= on Pith
- [52] Metacognition and cognitive monitoring: A new area of cognitive--developmental inquiry. , author=. American psychologist , volume=. 1979 , publisher= on Pith
- [53] Challenging big-bench tasks and whether chain-of-thought can solve them , author=. Findings of the Association for Computational Linguistics: ACL 2023 , pages= on Pith
- [54] Beyond the imitation game: Quantifying and extrapolating the capabilities of language models , author=. Transactions on machine learning research , year= on Pith
- [55] Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year= on PithPith's reviewarXiv
- [56] Openai gpt-5 system card , author=. arXiv preprint arXiv:2601.03267 , year= on PithPith's reviewarXiv
- [59] DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author= on Pith
- [60] OpenAI , title =. 2024 , url = on Pith
- [61] OpenAI , title =. 2025 , url = on Pith
- [62] Google DeepMind , title =. 2025 , url = on Pith
- [63] Sentence-bert: Sentence embeddings using siamese bert-networks , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages= on Pith
- [64] Dense passage retrieval for open-domain question answering , author=. Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages= on Pith
- [65] Metamemory: A theoretical framework and new findings , author=. Psychology of learning and motivation , volume=. 1990 , publisher= on Pith
- [66] Language models are few-shot learners , author=. Advances in neural information processing systems , volume= on Pith
- [67] A survey on large language model based autonomous agents , author=. Frontiers of Computer Science , volume=. 2024 , publisher= on Pith
- [68] Generative agents: Interactive simulacra of human behavior , author=. Proceedings of the 36th annual acm symposium on user interface software and technology , pages= on Pith
- [69] Moca: Measuring human-language model alignment on causal and moral judgment tasks , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [76] Skill-pro: Learning reusable skills from experience via non-parametric ppo for llm agents , author=. arXiv preprint arXiv:2602.01869 , year= on PithPith's reviewarXiv
- [77] SoK: Agentic Skills--Beyond Tool Use in LLM Agents , author=. arXiv preprint arXiv:2602.20867 , year= on PithPith's reviewarXiv
- [78] Evolver: Self-evolving llm agents through an experience-driven lifecycle , author=. arXiv preprint arXiv:2510.16079 , year= on PithPith's reviewarXiv
- [80] Meta-reasoner: Dynamic guidance for optimized inference-time reasoning in large language models , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages= on Pith
- [81] Metascale: Test-time scaling with evolving meta-thoughts , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages= on Pith
- [82] New embedding models and API updates , year = on Pith
- [83] Yasin Abbasi-Yadkori, D \'a vid P \'a l, and Csaba Szepesv \'a ri. 2011. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24 on Pith
- [84] Shipra Agrawal and Navin Goyal. 2013. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning, pages 127--135. PMLR on Pith
- [85] Marc-Antoine Allard, Arnaud Teinturier, Victor Xing, and Gautier Viaud. 2026. Experiential reflective learning for self-improving llm agents. arXiv preprint arXiv:2603.24639 on PitharXiv
- [86] Mingguang Chen, Licheng Wang, and Bo Qu. 2026. Recursive self-improvement in ai: From bounded self-refinement to autonomous research loops. arXiv preprint arXiv:2607.07663 on PithPith's reviewarXiv
- [87] Silin Chen, Shaoxin Lin, Xiaodong Gu, Yuling Shi, Heng Lian, Longfei Yun, Dong Chen, Weiguo Sun, Lin Cao, and Qianxiang Wang. 2025. Swe-exp: Experience-driven software issue resolution. arXiv preprint arXiv:2507.23361 on PitharXiv
- [88] Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. 2011. Contextual bandits with linear payoff functions. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 208--214. JMLR Workshop and Conference Proceedings on Pith
- [89] C Nicol \`o De Sabbata, Theodore R Sumers, Badr AlKhamissi, Antoine Bosselut, and Thomas L Griffiths. 2024. Rational metareasoning for large language models. arXiv preprint arXiv:2410.05563 on PithPith's reviewarXiv
- [90] DeepSeek-AI. 2026. Deepseek-v4: Towards highly efficient million-token context intelligence on Pith
- [91] Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E Primack, Summer Yue, and Chen Xing. 2025. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llm... on Pith
- [92] Emmanuel Dupoux, Yann LeCun, and Jitendra Malik. 2026. Why ai systems don't learn and what to do about it: Lessons on autonomous learning from cognitive science. arXiv preprint arXiv:2603.15381 on PitharXiv
- [93] Jonathan St BT Evans. 2011. Dual-process theories of reasoning: Contemporary issues and developmental applications. Developmental review, 31(2-3):86--102 on Pith
- [94] Jonathan St BT Evans and Keith E Stanovich. 2013. Dual-process theories of higher cognition: Advancing the debate. Perspectives on psychological science, 8(3):223--241 on Pith
- [95] Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu, Siwei Liu, Zihao Li, et al. 2025 a . A comprehensive survey of self-evolving ai agents: A new paradigm bridging foundation models and lifelong agentic systems. arXiv preprint arXiv:2508.07407 on PithPith's reviewarXiv
- [96] Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. 2025 b . Memp: Exploring agent procedural memory. arXiv preprint arXiv:2508.06433 on PithPith's reviewarXiv
- [97] Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al. 2025. A survey of self-evolving agents: On path to artificial super intelligence. arXiv preprint arXiv:2507.21046 on PithPith's reviewarXiv
- [98] Li Ji-An, Marcelo G Mattar, Hua-Dong Xiong, Marcus K Benna, and Robert C Wilson. 2025. Language models are capable of metacognitive monitoring and control of their internal activations. ArXiv, pages arXiv--2505 on Pith
- [99] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 on PithPith's reviewarXiv
- [100] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP)... on Pith
- [101] Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K Jain, Virginia Aglietti, Disha Jindal, Yuanzhu Peter Chen, et al. 2025. Big-bench extra hard. In Proceedings of the 63rd Annual Meeting of the Association for ... on Pith
- [102] Emre K c man, Robert Ness, Amit Sharma, and Chenhao Tan. 2023. Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:2305.00050 on PithPith's reviewarXiv
- [103] Wenjun Li, Dexun Li, Kuicai Dong, Cong Zhang, Hao Zhang, Weiwen Liu, Yasheng Wang, Ruiming Tang, and Yong Liu. 2025. Adaptive tool use in large language models with meta-cognition trigger. arXiv preprint arXiv:2502.12961 on PithPith's reviewarXiv
- [104] Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556 on PithPith's reviewarXiv
- [105] Jijia Liu, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang. 2023. Llm-powered hierarchical language agent for real-time human-ai coordination. arXiv preprint arXiv:2312.15224 on PithPith's reviewarXiv
- [106] Qin Liu, Wenxuan Zhou, Nan Xu, James Y Huang, Fei Wang, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2026. Metascale: Test-time scaling with evolving meta-thoughts. In Findings of the Association for Computational Linguistics: ACL 2026, pages 11828--11842 on Pith
- [107] Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch, and Peter Clark. 2023. Clin: A continually learning language agent for rapid task adaptation and generalization. arXiv preprint arXiv:2310.10134 on PithPith's reviewarXiv
- [108] Allen Nie, Yuhui Zhang, Atharva Shailesh Amdekar, Chris Piech, Tatsunori B Hashimoto, and Tobias Gerstenberg. 2024. Moca: Measuring human-language model alignment on causal and moral judgment tasks. Advances in Neural Information Processing Systems, 36 on Pith
- [109] OpenAI. 2024. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Gpt-4o mini: advancing cost-efficient intelligence on Pith
- [110] OpenAI . 2024. https://openai.com/index/new-embedding-models-and-api-updates/ New embedding models and api updates . January 25, 2024 on Pith
- [111] OpenAI. 2025. https://openai.com/index/gpt-5-system-card-update-gpt-5-2/ Update to gpt-5 system card: Gpt-5.2 on Pith
- [112] Siru Ouyang, Jun Yan, I Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T Le, Samira Daruki, Xiangru Tang, et al. 2025. Reasoningbank: Scaling agent self-evolving with reasoning memory. arXiv preprint arXiv:2509.25140 on PithPith's reviewarXiv
- [113] Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJC... on Pith
- [114] Yoan Russac, Claire Vernade, and Olivier Capp \'e . 2019. Weighted linear bandits for non-stationary environments. Advances in Neural Information Processing Systems, 32 on Pith
- [115] John Scoville, Shengzhuang Chen, Yejin Bang, Stefan Winzeck, and Jonathan Richard Schwarz. 2026. Cognitive demand steering for adaptive meta-reasoning in large language models. arXiv preprint arXiv:2608.01319 on PithPith's reviewarXiv
- [116] Kulin Shah, Nishanth Dikkala, Xin Wang, and Rina Panigrahy. 2024. Causal language modeling can elicit search and reasoning capabilities on logic puzzles. arXiv preprint arXiv:2409.10502 on PithPith's reviewarXiv
- [117] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634--8652 on Pith
- [118] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 on PithPith's reviewarXiv
- [119] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transac... on Pith
- [120] Yuan Sui, Yufei He, Tri Cao, Sophia Simeng Han, Yulin Chen, and Bryan Hooi. 2026. Meta-reasoner: Dynamic guidance for optimized inference-time reasoning in large language models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 13268--13286 on Pith
- [121] Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computati... on Pith
- [122] Xiangru Tang, Tianrui Qin, Tianhao Peng, Ziyang Zhou, Daniel Shao, Tingting Du, Xinming Wei, Peng Xia, Fang Wu, He Zhu, et al. 2025. Agent kb: Leveraging cross-domain experience for agentic problem solving. arXiv preprint arXiv:2507.06229 on PitharXiv
- [123] Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. 2026. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276 on PithPith's reviewarXiv
- [124] Gladys Tyen, Hassan Mansoor, Peter Chen, Tony Mak, and Victor C a rbune. 2023. Llms cannot find reasoning errors, but can correct them! arXiv preprint arXiv:2311.08516 on PithPith's reviewarXiv
- [125] Yuxuan Wan, Tianqing Fang, Zaitang Li, Yintong Huo, Wenxuan Wang, Haitao Mi, Dong Yu, and Michael R Lyu. 2026. Inference-time scaling of verification: Self-evolving deep research agents via test-time rubric-guided verification. arXiv preprint arXiv:2601.15808 on PithPith's reviewarXiv
- [126] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 on PithPith's reviewarXiv
- [127] Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H Chi, et al. 2025. Evo-memory: Benchmarking llm agent test-time learning with self-evolving memory. arXiv preprint arXiv:2511.20857 on PithPith's reviewarXiv
- [128] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. 2024. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314 on PithPith's reviewarXiv
- [129] Chunlong Wu and Zhibo Qu. 2025. Reflection-enhanced meta-optimization integrating textgrad-style prompt optimization with memory-driven self-evolution. arXiv preprint arXiv:2508.18749 on PithPith's reviewarXiv
- [130] Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. 2024. Os-copilot: Towards generalist computer agents with self-improvement. corr, abs/2402.07456, 2024. doi: 10.48550. arXiv preprint ARXIV.2402.07456 on PithPith's reviewarXiv
- [131] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110 on PithPith's reviewarXiv
- [132] Cheng Yang, Xuemeng Yang, Licheng Wen, Daocheng Fu, Jianbiao Mei, Rong Wu, Pinlong Cai, Yufan Shen, Nianchen Deng, Botian Shi, et al. 2025. Learning on the job: An experience-driven self-evolving agent for long-horizon tasks. arXiv preprint arXiv:2510.08002 on PitharXiv
- [133] Jenny Zhang, Bingchen Zhao, Wannan Yang, Jakob Foerster, Jeff Clune, Minqi Jiang, Sam Devlin, and Tatiana Shavrina. 2026 a . Hyperagents. arXiv preprint arXiv:2603.19461 on PithPith's reviewarXiv
- [134] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, et al. 2025 a . Agentic context engineering: Evolving contexts for self-improving language models. arXiv preprint arXiv:2510.04618 on PithPith's reviewarXiv
The first 100 of 104 references are listed here. All of them are in this paper's citation record.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment