REVIEW 4 major objections 5 minor 47 references
Tree-of-Experience: Hierarchical Experience Management for Self-Evolving Agents
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read An LLM agent that stores experience as a tree of reliability-weighted reasoning paths evolves faster and solves hard tasks better.
desk verdict A coherent framework with an honest appendix that reveals the Game of 24 result is mostly caching; FinEvolveBench needs independent evaluation before the headline claims are credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the experience tree $\mathcal{E} = \{(\Pi_i, Q_i, M_i)\}$: each node is an analytical perspective (in Game of 24, a canonicalized set of remaining numbers; in FinEvolveBench, an industry, a factor, or an analysis direction), each root-to-leaf path $\Pi_i$ is a full reasoning strategy, and each path carries a reliability state $Q_i$ in $[0, 2q_0]$ that environmental feedback raises or lowers after every use. The tree carries the whole framework: retrieval reranks the children of the current reasoning node and keeps the top-$k$ perspectives; a proposer invents new perspectives when none fit, subject to semantic subsumption, discriminative value, and balanced granularity; and maintenance merges similar nodes by embedding search plus an LLM judge, while reliability decay provides soft forgetting. The load-bearing update rule for stochastic environments is $Q(E_t) \leftarrow Q(E_t) + \eta \frac{1 - ((Q(E_t)-q_0)/q_0)^2}{1 + \log(1 + n_{\mathrm{hit}})} \cdot r_t$, which collapses to the binary rule $Q(e) = r_t \in \{0,1\}$ on Game of 24, where a deterministic verifier decides success after a single observation.
What would settle it
Two concrete checks settle the claim. On Game of 24, run the ToT baseline with the same verifier-confirmed solution-suffix cache that ToE uses (exact canonical-state memoization, with or without reliability weighting) and see whether the 85.3% versus 64.9% gap persists or collapses; the paper's appendix says failed continuations are excluded by design, which makes this comparison the crux. On FinEvolveBench, give the experience-free Pipe the same three-level industry–factor–analysis_direction decomposition without reliability-weighted retrieval, and also try inducing the hierarchy from reasoning traces by clustering instead of specifying it by hand; if the 41.24% average tsIC gain shrinks materially in either variant, the advantage is the pre-built structure rather than the calibration mechanism.
Extended reading notes
Core claim
The paper's central claim is that experience representations fail when they are disconnected from the reasoning process they are meant to aid, and that organizing experience as a hierarchy of analytical perspectives fixes this. In ToE, each stored experience is a root-to-leaf reasoning path whose reliability is tuned by outcome-level feedback, so a delayed success or failure can be propagated to the specific perspectives that produced it. The authors argue that this yields the four properties they set out: attributability of outcomes to reasoning components, transferability by reusing partial reasoning paths rather than whole task instances, evolvability through incremental calibration and merging of nodes, and efficiency through branch-limited retrieval. Empirically, on Game of 24 ToE reaches 85.3% accuracy against 64.9% for Tree-of-Thoughts while cutting LLM calls by 78.7%, and on FinEvolveBench it raises tsIC by an average of 41.24% across 12 evaluation settings, in a regime where conventional experience methods (Mem0, MemRL, ReMe) often underperform the experience-free pipeline. The paper also argues that traditional intra-trajectory and inter-trajectory experience methods fail specifically because they cannot attribute noisy outcome feedback, which can actively harm performance on low-repetition tasks.
Load-bearing premise
The reported gains assume that the experience-free baselines were not given any comparable structured memory, and on both benchmarks ToE enjoys a structural head start: in Game of 24 it reuses verifier-confirmed solution suffixes that ToT must rediscover from scratch, and in FinEvolveBench it is given a hand-built industry–factor–analysis hierarchy that matches the benchmark while the plain pipeline lacks any hierarchy at all.
Editorial extensions
If this is right
- On tasks with a well-defined reasoning structure and deterministic feedback (Game of 24), ToE raises accuracy from 64.9% to 85.3% over the experience-free ToT baseline and cuts average LLM calls per puzzle from 51.7 to 11.0, a 78.7% reduction.
- On low-repetition tasks with delayed, implicit outcome feedback (FinEvolveBench), ToE improves tsIC by 41.24% on average across 12 backbone–horizon–phase settings, with larger gains in the exploitation phase once the tree has stabilized.
- Conventional experience-management methods (Mem0, MemRL, ReMe) frequently underperform the experience-free baselines on both benchmarks, which the paper offers as evidence that mis-attributed experience can actively degrade reasoning.
- The framework is task-agnostic in structure: the same tree-plus-reliability machinery instantiates as verifier-confirmed state caching in Game of 24 and as a three-level industry–factor–analysis-direction hierarchy in financial sentiment.
- The formula-based reliability update outperforms an LLM-based judge in an ablation, suggesting that constrained, interpretable credit assignment beats free-form reflection when feedback is noisy.
Reading between the lines
- My reading: the Game of 24 result is largely a solved-suffix cache effect, because ToE stores verifier-confirmed continuations keyed by canonical number states, so any repeated or overlapping state short-circuits search. Equipping the ToT baseline with the same verified-suffix memoization would likely close much of the 20.4-point gap, and what remains would isolate the value of hierarchical attrib
- Because the FinEvolveBench hierarchy (industry to factor to analysis_direction) is manually specified to mirror the benchmark's own structure, a decisive test of the transfer claim would give the experience-free pipeline the same three-level decomposition without reliability-weighted retrieval, or learn the hierarchy from reasoning traces by clustering, as the paper says the initial hierarchy can
- The reliability scores suggest a natural next step the paper does not take: treating them as a cost-sensitive signal for deciding when to retrieve versus when to explore by proposing new perspectives, which would make the tree's growth itself adaptive to feedback.
- The 78.7% call reduction hints that ToE could serve as a general retrieval front-end for other structured-reasoning methods beyond the two tasks tested, though that generalization is my extrapolation, not the paper's claim.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Tree-of-Experience (ToE), a framework that organizes an LLM agent's accumulated experience as a hierarchical tree of analytical perspectives and reasoning paths, with reliability scores updated from environmental feedback. The method is evaluated on Game of 24 (with Tree of Thoughts as the reasoning backbone) and on FinEvolveBench, a financial sentiment-analysis benchmark. The authors report that ToE improves Game of 24 accuracy from 64.9% (ToT) to 85.3% while reducing LLM calls from 51.7 to 11.0 per puzzle, and improves tsIC by an average of 41.24% over the experience-free pipeline across 12 settings on FinEvolveBench. They also report that conventional memory methods (Mem0, MemRL, ReMe) often underperform experience-free baselines.
Significance. If the empirical results hold, ToE would be a valuable contribution to experience abstraction for LLM agents, combining structured representation, feedback attribution, and retrieval efficiency. The paper includes explicit prompt templates, implementation details, and a released code package, which are strengths for reproducibility. However, the current evidence is not sufficient to support the central claim because of a likely caching confound in the Game of 24 benchmark and missing uncertainty and reproducibility information in the FinEvolveBench evaluation. The core idea of aligning experience organization with the reasoning structure is interesting and worth further work, but the paper needs substantial additional experiments to substantiate its headline improvements.
major comments (4)
- [Appendix, Details in Game of 24] The Game of 24 implementation described in the Appendix is effectively a transposition table rather than a hierarchical experience-management system. The appendix states that a successful solution suffix is stored at a node and reused when the same or a sufficiently overlapping state is encountered again, and that retrieval first considers an exactly matched canonical number state. With eta_G24 = 1 (Eq. A4), the reliability update collapses to Q_new(e) = r_t in {0,1}, and only verifier-confirmed continuations are admitted. This means the ToE agent carries a verified-suffix cache across puzzles, while the experience-free ToT baseline (Section 'Performance on Game of 24', Table 2) searches from scratch on every puzzle. The reported 20.4-point accuracy gain and 78.7% call reduction could therefore be entirely explained by caching, rather than by the proposed hierarchical perspective alignment or reliability calibration. The authors should add a control baseline that gives ToT the same verified-suffix cache (e.g., a standard transposition table) and show that ToE still provides a meaningful gain beyond the cache.
- [Experimental Setup and Data availability and reproducibility] The FinEvolveBench results in Table 1 lack variance information: the paper states that each method is run three times and averaged, but no standard deviations, confidence intervals, or significance tests are reported. Given the stochastic nature of LLM predictions and the small differences in some settings (e.g., the 10-day Overall setting under Qwen3-35B-A3B where ToE is marginally worse than Pipe), the 41.24% average improvement is not statistically supported. Additionally, the released package does not include the raw news corpus and market-price files, so an independent researcher cannot reproduce the reported numbers without obtaining the (unidentified) data source. The authors should make the data available (or provide a public benchmark link) and report per-seed results or error bars.
- [Hyperparameters in FinEvolveBench and Formula-based experience-reliability update in FinEvolveBench] The main results depend on hyperparameters eta = 10, lambda = 0.8, k1 = 8, L = 3, and q0 = 1 with no sensitivity analysis. This is especially concerning for eta = 10 combined with reliability clipping to [0,2]: in Eq. (A6), the boundary factor and denominator produce an update of approximately 10 * r_t for first-use experiences (n_hit = 0), which saturates or zeroes the reliability after a single feedback signal. That is inconsistent with the text's description of incremental and graded calibration. A sensitivity analysis (e.g., eta in {1,3,10,30} and lambda in {0.5,0.8,1.0}) is needed to confirm the results are not artifacts of these fixed values.
- [Performance on FinEvolveBench and Tree Representation and Decoupled Execution] FinEvolveBench is authored by the same group (Deng et al. 2026), and for this benchmark the ToE hierarchy is manually specified as industry, factor, and analysis_direction, which mirrors the benchmark's internal structure. The comparison against Mem0 and MemRL therefore includes a structural advantage for ToE that is not a property of the framework itself. The paper should either evaluate ToE on a task where the hierarchy is induced automatically (as described in Section 'Granularity of Analytical Perspectives') or include a variant of ToE with an induced hierarchy on FinEvolveBench to disentangle the contribution of the prescribed hierarchy from the experience-management mechanism.
minor comments (5)
- [Ablation Study on Reliability Update Mechanisms] The phrase 'applies the update rule defined in Section .' is missing a section number; please cite the specific equation (e.g., Eq. 9 or Eq. A6).
- [Tables 1 and 2] Given that three independent runs are performed, the paper should report standard deviations or per-seed values in addition to the averages.
- [Eq. (9)] The notation Q(E_t) in [0,2q0]^|E_t| is confusing because the update rule is defined per experience e; please clarify whether Q is a vector of per-experience reliabilities and how the vectorized update is computed.
- [Granularity of Analytical Perspectives] The paper states that the experience tree is not a predefined template, but for FinEvolveBench the hierarchy is manually specified as a three-level structure; please reconcile this description with the manual construction used in the experiments.
- [Experimental Setup] The paper should state the number of Game of 24 puzzles and the number of news items and industries in FinEvolveBench, as these scale details are needed to assess the evaluation.
Circularity Check
No significant circularity: headline results are empirical benchmark measurements; the Game of 24 specialization is transparently a verifier-confirmed cache and FinEvolveBench is self-authored but externally falsifiable, neither reducing a prediction to its inputs.
full rationale
Walked the claimed derivation chain. The paper makes no first-principles derivation: all headline claims (85.3% vs 64.9% on Game of 24; 41.24% average tsIC improvement on FinEvolveBench) are empirical measurements from benchmark runs. The closest thing to a reduction is the Game of 24 appendix, where Eqs. (A3)-(A5) show that with eta_G24 = 1 the reliability update collapses to Q_new(e) = r_t in {0,1}, and retrieval is by exact canonical-state match or number-set overlap with verifier-confirmed suffixes. That is a transposition-table/memoization scheme, and the experience-free ToT baseline has no such cache, so the Game of 24 gain may partly be a caching effect; however, this is a baseline-comparison confound and an interpretation caveat, not a circular derivation, because the accuracy and call-count numbers are observed outcomes, not quantities constructed from the method's parameters. Similarly, FinEvolveBench is a self-authored benchmark (the 2026 citation has the same author list as the present paper), but it is a real-data, externally falsifiable evaluation corpus using standard IC metrics; the paper does not fit a parameter to its own outputs and then 'predict' those outputs. No equation equates a claimed prediction to an input by construction; no uniqueness theorem is invoked via self-citation; and the manually specified industry/factor/analysis_direction hierarchy is a design choice, not a hidden fit. The appendix explicitly acknowledges the binary collapse and the deterministic state-based specialization, so no load-bearing step is concealed. I therefore find no significant circularity.
Assumptions & free parameters
free parameters (6)
- eta (reliability learning rate) =
10
- q0 (initial reliability) =
1
- lambda (leaf retrieval threshold) =
0.8
- k1 (top-k candidates) =
8
- L (tree depth) =
3
- eta_G24 (Game of 24 update rate) =
1
assumptions (3)
- domain assumption The prompt-based LLM reranker f_rerank and proposer f_propose (Eqs. 3 and 4) reliably select and generate useful analytical perspectives.
- domain assumption The manually specified three-level hierarchy (industry, factor, analysis_direction) matches the causal structure of financial news sentiment for FinEvolveBench.
- domain assumption Delayed market outcomes (future returns) provide a valid per-experience credit-assignment signal when aggregated through Eq. A6.
invented entities (1)
-
Experience reliability score Q(e) (per-horizon impact values)
Cite this review
Pith. "Pith review of Tree-of-Experience: Hierarchical Experience Management for Self-Evolving Agents." pith.science (2026). https://pith.science/paper/UKWYEPEU
@misc{pith2026260809044,
author = {Pith},
title = {Pith review of: Tree-of-Experience: Hierarchical Experience Management for Self-Evolving Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/UKWYEPEU}},
note = {Machine review of arXiv:2608.09044}
}
read the original abstract
Continual self-evolution requires LLM agents to transform environmental interactions into reliable and reusable experience. Existing methods typically refine individual trajectories or abstract shared knowledge from related trajectories, but their experience representations are often disconnected from the underlying reasoning process. This limits feedback attribution, cross-task transfer, and update and retrieval efficiency, particularly in complex reasoning tasks with outcome-level feedback. To overcome this limitation, we propose \textbf{T}ree-\textbf{o}f-\textbf{E}xperience (ToE), a structured experience-management framework that aligns experience organization with the hierarchical reasoning process of LLM agents. Specifically, ToE organizes the experience into a shared tree of analytical perspectives and reasoning paths, whose reliability is calibrated through environmental outcomes to support systematic updating, transfer, and efficient retrieval. The experimental results on \textsc{Game of 24} and \textsc{FinEvolveBench} show that ToE substantially improves both problem-solving performance and efficiency. On \textsc{Game of 24}, ToE achieves a 31.4\% relative improvement in accuracy over the experience-free ToT baseline. On \textsc{FinEvolveBench}, ToE improves tsIC by an average of 41.24\% over the experience-free pipeline across 12 evaluation settings, whereas conventional experience-management methods often underperform experience-free baselines.
Figures
Reference graph
Works this paper leans on
-
[1]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Expel: Llm agents are experiential learners , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[2]
FinEvolveBench: A Benchmark for Self-Evolving Agents on Low-Repetition Tasks with Implicit Rewards
FinEvolveBench: A Benchmark for Self-Evolving Agents on Low-Repetition Tasks with Implicit Rewards , author=. arXiv preprint arXiv:2606.06960 , year=
-
[3]
arXiv preprint arXiv:2501.14249 , year=
Humanity's last exam , author=. arXiv preprint arXiv:2501.14249 , year=
-
[4]
arXiv preprint arXiv:2010.03768 , year=
Alfworld: Aligning text and embodied environments for interactive learning , author=. arXiv preprint arXiv:2010.03768 , year=
arXiv 2010
-
[5]
International Conference on Learning Representations , volume=
Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions , author=. International Conference on Learning Representations , volume=
- [6]
-
[7]
Companion Proceedings of the Web Conference , year =
Maia, Macedo and Handschuh, Siegfried and Freitas, Andr. Companion Proceedings of the Web Conference , year =
-
[8]
Proceedings of the Italian Conference on Computational Linguistics , year =
Benchmarking Large Language Models for Target-Based Financial Sentiment Analysis , author =. Proceedings of the Italian Conference on Computational Linguistics , year =
Show all 47 references
-
[9]
Advances in Neural Information Processing Systems , year =
FinBen: A Holistic Financial Benchmark for Large Language Models , author =. Advances in Neural Information Processing Systems , year =
-
[10]
and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , journal =
Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , journal =
-
[11]
and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , journal =
Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , journal =
- [12]
-
[13]
arXiv preprint arXiv:2506.13651 , year =
XBench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations , author =. arXiv preprint arXiv:2506.13651 , year =
-
[14]
arXiv preprint arXiv:2506.07982 , year =
^2 -Bench: Evaluating Conversational Agents in a Dual-Control Setting , author =. arXiv preprint arXiv:2506.07982 , year =
-
[15]
Wu, Shijie and Irsoy, Ozan and Lu, Steven and others , journal =
-
[16]
arXiv preprint arXiv:2402.02315 , year =
A Survey of Large Language Models in Finance , author =. arXiv preprint arXiv:2402.02315 , year =
-
[17]
arXiv preprint arXiv:2406.11903 , year =
A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges , author =. arXiv preprint arXiv:2406.11903 , year =
-
[18]
Chen, Zhiyu and Chen, Wenhu and Smiley, Charese and Shah, Sameena and Borova, Iana and Langdon, Dylan and Moussa, Reema and Beane, Matt and Huang, Ting-Hao and Routledge, Bryan and Wang, William Yang , journal =
-
[19]
Zhu, Fengbin and Lei, Wenqiang and Huang, Youcheng and Wang, Chao and Zhang, Shuo and Lv, Jiancheng and Feng, Fuli and Chua, Tat-Seng , booktitle =
-
[20]
Chen, Yanxu and Yao, Zijun and Liu, Yantao and Xin, Amy and Ye, Jin and Yu, Jianing and Hou, Lei and Li, Juanzi , journal =
-
[21]
Advances in Neural Information Processing Systems , volume =
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks , author =. Advances in Neural Information Processing Systems , volume =
-
[22]
Advances in Neural Information Processing Systems , volume =
Reflexion: Language Agents with Verbal Reinforcement Learning , author =. Advances in Neural Information Processing Systems , volume =
-
[23]
2025 , eprint =
Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory , author =. 2025 , eprint =
2025
-
[24]
2025 , eprint =
LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners , author =. 2025 , eprint =
2025
-
[25]
arXiv preprint arXiv:2511.20857 , year =
Evo-Memory: Benchmarking LLM Agent Test-Time Learning with Self-Evolving Memory , author =. arXiv preprint arXiv:2511.20857 , year =
-
[26]
Google AI , year =
Welcome to the Era of Experience , author =. Google AI , year =
-
[27]
Zhang, Shengtao and Wang, Jiaqian and Zhou, Ruiwen and Liao, Junwei and Feng, Yuchen and others , journal =
-
[28]
Araci, Dogu , journal =
-
[29]
Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence , year =
Deep Learning for Event-Driven Stock Prediction , author =. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence , year =
-
[30]
Lopez-Lira, Alejandro and Tang, Yuehua , journal =. Can
-
[31]
Yang, Hongyang and Liu, Xiao-Yang and Wang, Christina Dan , journal =
-
[32]
Wang, Mengyu and Ma, Tiejun , booktitle =
-
[33]
2019 IEEE Fifth International Conference on Big Data Computing Service and Applications , pages =
Stock Price Prediction Using News Sentiment Analysis , author =. 2019 IEEE Fifth International Conference on Big Data Computing Service and Applications , pages =
2019
-
[34]
Neurocomputing , volume =
Combining the Wisdom of Crowds and Technical Analysis for Financial Market Prediction Using Deep Random Subspace Ensembles , author =. Neurocomputing , volume =
-
[35]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Skillgen: Learning domain skills for in-context sequential decision making , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[36]
arXiv preprint arXiv:2511.06449 , year=
Flex: Continuous agent evolution via forward learning from experience , author=. arXiv preprint arXiv:2511.06449 , year=
-
[37]
arXiv preprint arXiv:2602.08234 , year=
Skillrl: Evolving agents via recursive skill-augmented reinforcement learning , author=. arXiv preprint arXiv:2602.08234 , year=
-
[38]
International Conference on Learning Representations , volume=
Synapse: Trajectory-as-exemplar prompting with memory for computer control , author=. International Conference on Learning Representations , volume=
-
[39]
Advances in Neural Information Processing Systems , volume=
Vlm agents generate their own memories: Distilling experience into embodied programs of thought , author=. Advances in Neural Information Processing Systems , volume=
-
[40]
Advances in Neural Information Processing Systems , volume=
Large language models are semi-parametric reinforcement learning agents , author=. Advances in Neural Information Processing Systems , volume=
-
[41]
International Conference on Learning Representations , volume=
Agent s: An open agentic framework that uses computers like a human , author=. International Conference on Learning Representations , volume=
-
[42]
Findings of the Association for Computational Linguistics: ACL 2026 , pages=
Remember me, refine me: A dynamic procedural memory framework for experience-driven agent evolution , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=
2026
-
[43]
Advances in Neural Information Processing Systems , volume=
Generalizing experience for language agents with hierarchical metaflows , author=. Advances in Neural Information Processing Systems , volume=
-
[44]
Advances in Neural Information Processing Systems , volume=
G-memory: Tracing hierarchical memory for multi-agent systems , author=. Advances in Neural Information Processing Systems , volume=
-
[45]
The Twelfth International Conference on Learning Representations , year=
Chain of thought empowers transformers to solve inherently serial problems , author=. The Twelfth International Conference on Learning Representations , year=
-
[46]
Advances in neural information processing systems , volume=
Tree of thoughts: Deliberate problem solving with large language models , author=. Advances in neural information processing systems , volume=
-
[47]
Advances in neural information processing systems , volume=
Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.