Pith. sign in

REVIEW 3 major objections 5 minor 64 references

Disentangling Exploration of Large Language Models by Optimal Exploitation

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that comparing an LLM agent's return with the return of an optimal exploitation oracle isolates how much of its shortfall comes from missing exploration rather than weak exploitation.

desk verdict A clean, useful regret decomposition for isolating LLM exploration, but the oracle it rests on is under-validated and its description is internally inconsistent. read the letter →

arxiv 2501.08925 v3 pith:U4EMEGOO submitted 2025-01-15 cs.LG cs.AIcs.CL

Tim Grams , Patrick Betz , Sascha Marton , Stefan Lüdtke , Christian Bartelt This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords explorationexploitationregretdecompositionlargelanguagemodelsin-contextreinforcementlearningoptimaloracleLLMagentsstatespacecoverage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that exploration by LLM agents can be evaluated on its own, without being tangled up with exploitation. It proposes decomposing an agent's cumulative regret into an exploration gap and an exploitation gap using the return of an optimal exploitation oracle that acts only on the agent's memory. In deterministic grid and maze environments, the exploration gap is zero exactly when the agent has gathered the information needed for maximal return, so any remaining gap is attributed purely to missing exploration. Experiments with several open and closed LLMs show that most models explore worse than a random policy and that exploration gaps shrink as general reasoning ability, measured by MMLU-Pro score, rises. The authors argue this decomposition is needed because agent returns and LLM-based exploitation estimates do not reliably reflect exploration progress.

What carries the argument

The central object is the optimal exploitation oracle $\sigma_{\text{exploit}}$, a predefined policy that, given the agent's history, selects the locally reward-maximizing action at every step and is used as a diagnostic tool rather than as a trainable policy. In the authors' environments the oracle is implemented by solving an orienteering problem over the reward locations the agent has seen. The identity that carries the argument is the regret decomposition $\text{Regret} = \Delta_{\text{explore}} + \Delta_{\text{exploit}}$, which turns the exploration gap into a direct measure of missing information about the environment.

What would settle it

Find any environment instance where the oracle's optimal exploitation path, computed from the agent's memory, is beaten by a different path using the same memory: that would show the oracle is not exact, and the decomposition would misattribute exploitation error to exploration. Concretely, enumerate all reachable paths from the agent's observations in a small maze and compare the oracle's return to the true maximum; a single mismatch invalidates the decomposition's interpretation.

Watch

Extended reading notes

Core claim

The central claim is that the optimal exploitation return provides an accurate disentanglement: by calculating how much an optimal exploiter could earn from the agent's memory, one can measure exploration without conflating it with exploitation. Formally, letting $R^*$ be the maximal achievable return, $R^{\sigma_{\text{exploit}}}$ be the return of the optimal exploitation oracle, and $R$ be the agent's return, the regret decomposes as $R^* - R = (R^* - R^{\sigma_{\text{exploit}}}) + (R^{\sigma_{\text{exploit}}} - R)$, where the first parenthesized term is the exploration gap and the second is the exploitation gap. Because the oracle uses only gathered information, a positive exploration gap indicates that the agent did not explore enough, and a gap of zero proves that its information suffices for optimal return. The paper then applies this metric to LLM agents in symbolic Treasure Rooms and maze environments and finds that most models fail to reach full coverage or near-zero exploration gaps.

Load-bearing premise

The framework assumes the optimal exploitation oracle is truly optimal and uses only the information the agent actually gathered, so the exploration gap is exactly the return lost to missing information; if the oracle is approximate or leaks knowledge of unvisited rewards, the gap is biased.

Editorial extensions

If this is right

  • Agent return alone cannot rank exploration: models with similar returns can have significantly different exploration gaps, and lower-return agents can be exploring more effectively.
  • Model comparisons should be based on optimal exploitation return rather than raw return or LLM-based exploitation estimates, because those measures conflate the two skills.
  • Weak LLMs are not sufficient for exploration; stronger reasoning models explore better, so exploration-heavy agent design should allocate the task to stronger models.
  • Prompt engineering effects on exploration can be measured directly, as task-oriented, undirected, and exploitation-prompts change the balance between the exploration and exploitation gaps.
  • Exploration performance degrades in longer-horizon environments, so long-horizon planning is a key bottleneck for LLM exploration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The exactness of the decomposition hangs on the oracle being truly optimal with only the agent's observations; if the implemented orienteering solver is approximate, the reported exploration gaps would be overestimates and the exploitation gaps underestimates.
  • The MMLU-Pro correlation suggests exploration is mediated by general planning and reasoning ability, but the paper does not establish a causal direction; controlled manipulation of reasoning traces could test this.
  • The same decomposition could be extended to stochastic environments by replacing the deterministic maximal return with an expected optimal return, though the identity would then require probability-aware treatment.
  • The metric offers a direct testbed for memory design, such as comparing reflection-based or compressed histories, to see whether reducing memory redundancy improves the exploration gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an evaluation framework for disentangling exploration from exploitation in LLM-based agents operating in deterministic, partially observable environments. The central idea is to define an optimal exploitation oracle σ_exploit that, given the agent's accumulated memory, computes the best possible return from the information gathered. The exploration gap Δ_explore is then defined as R* − R^σ, where R* is the maximal achievable return, and the exploitation gap as R^σ − R_agent, so that cumulative regret decomposes exactly as Δ_explore + Δ_exploit. The authors apply this decomposition to Treasure Rooms and Maze environments, compare eleven LLMs plus a random baseline, report that most models leave large exploration gaps, and correlate exploration performance with MMLU-Pro scores. They also use the decomposition to analyze how different prompting instructions change exploration versus exploitation behavior. The manuscript is an arXiv preprint that defers several details, including the oracle implementation and an ablation, to a self-cited 'full version' reference.

Significance. If the optimal exploitation oracle is valid, the proposed decomposition is a conceptually clean and useful diagnostic tool: it gives a scalar measure of exploration progress that is not confounded by the agent's ability to exploit known information, and the empirical study provides one of the first systematic LLM evaluations with exploration isolated from exploitation. The paper honestly acknowledges that the oracle is a strong assumption, and it includes a useful set of controlled environments, a broad model comparison, and a prompt-engineering case study. The main contribution is therefore a measurement methodology rather than a new learning algorithm, and its value depends entirely on whether the oracle can be computed exactly and without using information the agent has not actually gathered.

major comments (3)
  1. [Section 3.2 and Section 3.3] The definition of the oracle is internally inconsistent. Section 3.2 defines σ_exploit as 'optimally select[ing] the locally reward-maximizing action at every time step,' which describes a greedy policy, whereas Section 3.3 states the oracle is implemented by 'search[ing] for the path between balls with the highest reward,' which is a global orienteering problem. A local greedy policy and a globally optimal route are not the same: in the maze and Treasure Rooms environments, an optimal path may pass through low-reward balls to reach a high-reward ball, and greedy per-step maximization can fail. This distinction is load-bearing because Δ_explore = R* − R^σ isolates exploration only if σ is exactly optimal given the agent's memory. If the implemented oracle is greedy or approximate, then R^σ < R* even with complete state coverage, and the exploration gap is contaminated by exploitation suboptimality. The main text must specify the exact optimization problem solved, prove or empirically validate that the implementation attains the optimum, and demonstrate that the oracle uses only the agent's observed memory rather than the unvisited reward layout.
  2. [Appendix A.6 / reference [15]] The central implementation is not actually contained in the manuscript. The text says 'See more details in Appendix A.6 [15]' and reference [15] is the authors' own arXiv full version of this same paper, so the main text is not self-contained. There is no pseudocode, no correctness statement, no complexity bound, and no validation of the oracle in the main text. The ablation in Appendix A.7 tests whether an LLM can serve as a replacement for the oracle, but it does not test whether the actual oracle is exact. Since Tables 1-2 and Figure 5 are all computed through this oracle, the main empirical claims rest on an unverified component that the reader cannot inspect in the submitted manuscript. The authors should provide the oracle algorithm and a validation study (for example, comparison against brute-force optimal solutions on small instances) directly in the main text or in an accessible appendix.
  3. [Section 3.2, 'On the complexity of defining an optimal exploitation'] The paper admits that the optimal exploitation assumption is 'strong' and that 'approximators are an option, but should be consistent; otherwise, they might introduce ambiguity.' This is an explicit acknowledgement that the framework's accuracy depends on oracle correctness, but no sensitivity analysis is provided. A concrete test would be to compute R^σ with an exact solver on the smallest environments and compare it against the oracle used for the reported results, or to perturb the oracle's optimality and show that model rankings are stable. Without such a test, the reader cannot know whether the reported differences between models (for example, o1-mini versus Gemini 1.5 Pro in Table 1) are due to exploration ability or to differential sensitivity to a suboptimal oracle.
minor comments (5)
  1. [Table 1] In the Claude 3.5 Sonnet row, the Regret value for the 'Last Episode' column is printed as '41%' while all other Regret entries are decimals; this should be '0.41' for consistency.
  2. [Section 3.3] The sentence 'The optimal exploitation oracle is fastest by solving an instance of the orienteering optimization problem' appears to contain a typo; it should likely read 'is implemented by solving' or 'is obtained by solving.'
  3. [Figure 4] The vertical axis label 'Optimal Return − Δ explore' is confusing because the text says the figure regresses exploration gaps against MMLU-Pro; please clarify what is plotted (for example, 'R* − Δ_explore' or 'normalized exploration gap').
  4. [Section 4.2] The definition of sample efficiency as 'converge to the 90% equilibrium of the maximum achieved exploitation return' is vague; please define 'equilibrium' precisely and state how the 90% threshold is computed over episodes.
  5. [Reference [15]] The paper cites its own full version as reference [15] with the same arXiv identifier as the manuscript itself; this self-reference is unusual and should be explained or the relevant content should be included in the submission.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the regret decomposition is an algebraic identity built on an external optimal-exploitation oracle; self-citations to the full version are supplementary and not load-bearing.

full rationale

The central derivation is Eqs. (2)-(4): the exploration gap is R* minus the oracle return R^sigma, the exploitation gap is R^sigma minus the agent return, and the paper states that 'it is easy to see that it holds by construction: Regret_i(T) = Delta_explore_i + Delta_exploit_i.' This is an algebraic identity once R^sigma is defined as the return of an optimal policy conditioned on the agent's history. The exploration gap is not fitted to agent behavior or to a prediction target; it is computed from an externally defined oracle, so it is not a fitted input called a prediction. The assertion that a nonzero gap can only be caused by a lack of exploration is a conditional assumption about the oracle's optimality and the environment's determinism, not a circular step. The paper's self-citations to [15] are for implementation details, layouts, prompts, and full result tables in the appendices; none of these citations justifies the derivation itself, and no uniqueness theorem or prior result by the same authors is invoked to forbid alternatives. The Appendix A.7 ablation substitutes an LLM for the oracle and is presented as evidence that LLM exploitation is unreliable; that is an empirical control, not an input to the metric. One genuine caveat, which is a correctness risk rather than circularity, is that Section 3.2 defines the oracle as selecting the 'locally reward-maximizing action' while Section 3.3 describes solving a global orienteering problem, and the exactness of the implemented oracle is not validated in the main text. If the oracle is approximate, Delta_explore would be biased, but the decomposition would remain an identity relative to the implemented oracle. Under the hard rules, an unvalidated or internally inconsistent oracle is not circular reasoning, so the score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The metric itself has no fitted parameters: the decomposition is definitional. The oracle, determinism, and memory assumptions carry the analytic weight, and the MMLU-Pro correlation is a post hoc statistical fit. The oracle is the main invented construct.

free parameters (3)
  • MMLU-Pro versus exploration gap linear regression slope = not reported
    Fitted to 11 model-level averages to support the claim that exploration correlates with reasoning; the slope value is not given in the main text.
  • Sample efficiency 90% equilibrium threshold = 90% of maximum achieved exploitation return
    Arbitrary threshold in Section 4.2; affects sample-efficiency comparisons but not the central decomposition.
  • Episode step budget = distance from start to furthest room per layout
    Environment difficulty hyperparameter chosen in Section 3.3; results such as coverage and gaps depend on it, but it is not fitted to make the decomposition work.
assumptions (4)
  • domain assumption The environment is a deterministic MDP with fixed layout and hidden ball rewards.
    Section 3.1 formalizes a deterministic MDP; the inequality chain R* >= R^oracle >= Ri relies on determinism.
  • ad hoc to paper The optimal exploitation oracle can be computed and uses only the agent's memory, not unobserved rewards.
    Section 3.2 defines the oracle; the entire exploration gap rests on this premise. The paper itself calls it a strong assumption.
  • domain assumption Visiting a room reveals the ball reward, and the agent's memory retains all history without loss.
    Environment design in Section 3.3; needed for state space coverage to imply knowledge of rewards.
  • domain assumption MMLU-Pro scores are a valid proxy for reasoning capability.
    Used in Section 4.4 to interpret the exploration correlation as a reasoning correlation.
invented entities (1)
  • Optimal exploitation oracle
    purpose: Diagnostic baseline that computes the best return achievable from the agent's current memory, used to define exploration and exploitation gaps.
    A methodological construct introduced by the paper, not a physical entity. Its validity rests on the paper's implementation, which is deferred to an appendix, so it has no external falsifiable handle in this preprint.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disentangling Exploration of Large Language Models by Optimal Exploitation." pith.science (2026). https://pith.science/paper/U4EMEGOO

@misc{pith2026250108925,
  author       = {Pith},
  title        = {Pith review of: Disentangling Exploration of Large Language Models by Optimal Exploitation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U4EMEGOO}},
  note         = {Machine review of arXiv:2501.08925}
}
read the original abstract

Exploration is a crucial skill for in-context reinforcement learning in unknown environments. However, it remains unclear if large language models can effectively explore a partially hidden state space. This work isolates exploration as the sole objective, tasking an agent with gathering information that enhances future returns. Within this framework, we argue that measuring agent returns is not sufficient for a fair evaluation. Hence, we decompose missing rewards into their exploration and exploitation components based on the optimal achievable return. Experiments with various models reveal that most struggle to explore the state space, and weak exploration is insufficient. Nevertheless, we found a positive correlation between exploration performance and reasoning capabilities. Our decomposition can provide insights into differences in behaviors driven by prompt engineering, offering a valuable tool for refining performance in exploratory tasks.

Figures

Figures reproduced from arXiv: 2501.08925 by the authors.

Figure 1
Figure 1. An overview of our evaluation procedure based on an optimal exploitation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Exploration is independent of the episode return. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The agent return, exploration gap, and grid coverage in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Exploration gaps strongly correlate with knowledge and reasoning capabilities. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The normalized mean regret decomposed into exploration and exploitation components. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 22 canonical work pages

  1. [15]

    Disentangling Exploration of Large Language Models by Optimal Exploitation

    T. Grams, P. Betz, S. Marton, S. Lüdtke, and C. Bartelt. Disentangling exploration of large language models by optimal exploitation. arXiv preprint arXiv:2501.08925, 2025. Full version of this paper

    work page Pith review arXiv 2025

  2. [1]

    Achiam and S

    J. Achiam and S. Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv:1703.01732, 2017

    arXiv 2017

  3. [2]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 tech- nical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [3]

    M. AI. Mistral small. https://mistral.ai/news/september-24-release/,

    work page

  5. [4]

    M. AI. Mistral large 2i. https://mistral.ai/news/mistral-large-2407/,

    work page

  6. [5]

    Claude 3.5 Sonnet

    Anthrophic. Claude 3.5 Sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet, 2024. [Accessed 10-01-2025]

    work page 2024

  7. [6]

    [Accessed 10-01-2025]

    work page 2025

  8. [7]

    Bellemare, S

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016

    2016

Show all 64 references
  1. [8]

    Avner, S

    O. Avner, S. Mannor, and O. Shamir. Decoupling exploration and exploitation in multi-armed bandits. arXiv preprint arXiv:1205.2874 , 2012

    2012 arXiv

  2. [9]

    D. Chen, Q. Zhang, and Y . Zhu. Efficient sequential decision making with large language models. arXiv preprint arXiv:2406.12125, 2024

    2024 arXiv

  3. [10]

    Burda, H

    Y . Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by ran- dom network distillation. arXiv preprint arXiv:1810.12894, 2018

    2018 arXiv

  4. [11]

    Chevalier-Boisvert, B

    M. Chevalier-Boisvert, B. Dai, M. Towers, R. Perez-Vicente, L. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environ- ments for goal-oriented tasks. Advances in Neural Information Pro- cessing System...

    2024

  5. [12]

    Chevalier-Boisvert, D

    M. Chevalier-Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Sa- haria, T. H. Nguyen, and Y . Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272, 2018

    2018 arXiv

  6. [13]

    Dubey, A

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [14]

    M.-A. Côté, A. Kádár, X. Yuan, B. Kybartas, T. Barnes, E. Fine, J. Moore, M. Hausknecht, L. El Asri, M. Adada, et al. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference o...

    2018

  8. [16]

    Ecoffet, J

    A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. Go- explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995, 2019

    1901 arXiv

  9. [17]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  10. [18]

    Grigsby, L

    J. Grigsby, L. Fan, and Y . Zhu. Amago: Scalable in-context reinforce- ment learning for adaptive agents. arXiv preprint arXiv:2310.09971 , 2023

    2023 arXiv

  11. [19]

    S. Hao, Y . Gu, H. Ma, J. J. Hong, Z. Wang, D. Z. Wang, and Z. Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023

    2023 arXiv

  12. [20]

    D. Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06780, 2021

    2021 arXiv

  13. [21]

    Huang, W

    X. Huang, W. Liu, X. Chen, X. Wang, D. Lian, Y . Wang, R. Tang, and E. Chen. Wese: Weak exploration to strong exploitation for llm agents. arXiv preprint arXiv:2404.07456, 2024

    2024 arXiv

  14. [22]

    Huang, X

    J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023

    2023 arXiv

  15. [23]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  16. [24]

    Imani, L

    S. Imani, L. Du, and H. Shrivastava. Mathprompter: Mathematical rea- soning using large language models. arXiv preprint arXiv:2303.05398, 2023

    2023 arXiv

  17. [25]

    N. R. Ke, D. P. Sawyer, H. Soyer, M. Engelcke, D. P. Reichert, D. A. Hudson, J. Reid, A. Lerchner, D. J. Rezende, T. P. Lillicrap, et al. Can foundation models actively gather information in interactive environ- ments to test hypotheses? arXiv preprint arXiv:2412.06438, 2024

    2024

  18. [26]

    Kambhampati, K

    S. Kambhampati, K. Valmeekam, L. Guan, M. Verma, K. Stechly, S. Bhambri, L. Saldyt, and A. Murthy. Llms can’t plan, but can help planning in llm-modulo frameworks. arXiv preprint arXiv:2402.01817, 2024

    2024 arXiv

  19. [27]

    Krishnamurthy, K

    A. Krishnamurthy, K. Harris, D. J. Foster, C. Zhang, and A. Slivkins. Can large language models explore in-context? arXiv preprint arXiv:2403.15371, 2024

    2024 arXiv

  20. [28]

    A. S. Klyubin, D. Polani, and C. L. Nehaniv. Empowerment: A univer- sal agent-centric measure of control. In 2005 ieee congress on evolu- tionary computation, volume 1, pages 128–135. IEEE, 2005

    2005

  21. [29]

    Laskin, L

    M. Laskin, L. Wang, J. Oh, E. Parisotto, S. Spencer, R. Steiger- wald, D. Strouse, S. Hansen, A. Filos, E. Brooks, et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215, 2022

    2022 arXiv

  22. [30]

    Küttler, N

    H. Küttler, N. Nardelli, A. Miller, R. Raileanu, M. Selvatici, E. Grefen- stette, and T. Rocktäschel. The nethack learning environment.Advances in Neural Information Processing Systems, 33:7671–7684, 2020

    2020

  23. [31]

    Lillicrap

    T. Lillicrap. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  24. [32]

    X. L. Li, A. Kuncoro, J. Hoffmann, C. d. M. d’Autume, P. Blunsom, and A. Nematzadeh. A systematic investigation of commonsense knowl- edge in large language models. arXiv preprint arXiv:2111.00607, 2021

    2021 arXiv

  25. [33]

    C. Lu, S. Hu, and J. Clune. Intelligent go-explore: Standing on the shoulders of giant foundation models. arXiv preprint arXiv:2405.15143, 2024

    2024 arXiv

  26. [34]

    X. Liu, H. Yu, H. Zhang, Y . Xu, X. Lei, H. Lai, Y . Gu, H. Ding, K. Men, K. Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688, 2023

    2023 arXiv

  27. [35]

    Mazoure, J

    B. Mazoure, J. Bruce, D. Precup, R. Fergus, and A. Anand. Accelerating exploration and representation learning with offline pre-training. arXiv preprint arXiv:2304.00046, 2023

    2023 arXiv

  28. [36]

    K. Ma, H. Zhang, H. Wang, X. Pan, W. Yu, and D. Yu. Laser: Llm agent with state-space exploration for web navigation. arXiv preprint arXiv:2309.08172, 2023

    2023 arXiv

  29. [37]

    A. Nie, Y . Su, B. Chang, J. N. Lee, E. H. Chi, Q. V . Le, and M. Chen. Evolve: Evaluating and optimizing llms for exploration. arXiv preprint arXiv:2410.06238, 2024

    2024 arXiv

  30. [38]

    V . Mnih. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  31. [39]

    OpenAI o1-mini

    OpenAI. OpenAI o1-mini. https://platform.openai.com/docs/models# o1, 2024. [Accessed 10-01-2025]

    2024

  32. [40]

    Norman and J

    B. Norman and J. Clune. First-explore, then exploit: Meta-learning in- telligent exploration. arXiv preprint arXiv:2307.02276, 2023

    2023 arXiv

  33. [41]

    C. Park, X. Liu, A. Ozdaglar, and K. Zhang. Do llm agents have regret? a case study in online learning and games. arXiv preprint arXiv:2403.16843, 2024

    2024

  34. [42]

    Paglieri, B

    D. Paglieri, B. Cupiał, S. Coward, U. Piterbarg, M. Wolczyk, A. Khan, E. Pignatelli, Ł. Kuci ´nski, L. Pinto, R. Fergus, et al. Balrog: Bench- marking agentic llm and vlm reasoning on games. arXiv preprint arXiv:2411.13543, 2024

    2024 arXiv

  35. [43]

    S. K. Paul. Sequential planning in large partially observable environ- ments guided by llms. arXiv preprint arXiv:2312.07368, 2023

    2023 arXiv

  36. [44]

    Pathak, P

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778–2787. PMLR, 2017

    2017

  37. [45]

    Ruoss, F

    A. Ruoss, F. Pardo, H. Chan, B. Li, V . Mnih, and T. Genewein. Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations. arXiv preprint arXiv:2412.01441, 2024

    2024 arXiv

  38. [46]

    W. T. Piriyakulkij, C. Langenfeld, T. A. Le, and K. Ellis. Doing experi- ments and revising rules with natural language and probabilistic reason- ing. arXiv preprint arXiv:2402.06025, 2024

    2024 arXiv

  39. [47]

    Schäfer, F

    L. Schäfer, F. Christianos, J. Hanna, and S. V . Albrecht. Decoupling exploration and exploitation in reinforcement learning. In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021

    2021

  40. [48]

    Samvelyan, R

    M. Samvelyan, R. Kirk, V . Kurin, J. Parker-Holder, M. Jiang, E. Ham- bro, F. Petroni, H. Küttler, E. Grefenstette, and T. Rocktäschel. Mini- hack the planet: A sandbox for open-ended reinforcement learning re- search. arXiv preprint arXiv:2109.13202, 2021

    2021 arXiv

  41. [49]

    Schmied, F

    T. Schmied, F. Paischer, V . Patil, M. Hofmarcher, R. Pascanu, and S. Hochreiter. Retrieval-augmented decision transformer: External memory for in-context rl. arXiv preprint arXiv:2410.07071, 2024

    2024 arXiv

  42. [50]

    Schmidhuber

    J. Schmidhuber. Formal theory of creativity, fun, and intrinsic motiva- tion (1990–2010). IEEE transactions on autonomous mental develop- ment, 2(3):230–247, 2010

    1990

  43. [51]

    Y . Song, W. Xiong, X. Zhao, D. Zhu, W. Wu, K. Wang, C. Li, W. Peng, and S. Li. Agentbank: Towards generalized llm agents via fine-tuning on 50000+ interaction trajectories. arXiv preprint arXiv:2410.07706 , 2024

    2024 arXiv

  44. [52]

    Shinn, F

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflex- ion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  45. [53]

    G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupati- raju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv. org/abs/2408.00118, 1(3), 2024

    2024 arXiv

  46. [54]

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  47. [55]

    Y . Wang, X. Ma, G. Zhang, Y . Ni, A. Chandra, S. Guo, W. Ren, A. Arul- raj, X. He, Z. Jiang, et al. Mmlu-pro: A more robust and challeng- ing multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benc...

    2024

  48. [56]

    Valmeekam, M

    K. Valmeekam, M. Marquez, S. Sreedharan, and S. Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36:75993–76005, 2023

    2023

  49. [57]

    Y . Wu, X. Tang, T. M. Mitchell, and Y . Li. Smartplay: A benchmark for llms as intelligent agents. arXiv preprint arXiv:2310.01557, 2023

    2023 arXiv

  50. [58]

    W. F. Whitney, M. Bloesch, J. T. Springenberg, A. Abdolmaleki, K. Cho, and M. Riedmiller. Decoupled exploration and exploitation policies for sample-efficient reinforcement learning. arXiv preprint arXiv:2101.09458, 2021

    2021 arXiv

  51. [59]

    S. Yao, H. Chen, J. Yang, and K. Narasimhan. Webshop: Towards scal- able real-world web interaction with grounded language agents. Ad- vances in Neural Information Processing Systems , 35:20744–20757, 2022

    2022

  52. [60]

    F. Xu, Q. Lin, J. Han, T. Zhao, J. Liu, and E. Cambria. Are large lan- guage models really good logical reasoners? a comprehensive evalu- ation from deductive, inductive and abductive views. arXiv preprint arXiv:2306.09841, 2023

    2023 arXiv

  53. [61]

    Z. Yuan, H. Yuan, C. Li, G. Dong, K. Lu, C. Tan, C. Zhou, and J. Zhou. Scaling relationship on learning mathematical reasoning with large lan- guage models. arXiv preprint arXiv:2308.01825, 2023

    2023 arXiv

  54. [62]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  55. [63]

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

  56. [64]

    A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y . Dong, and J. Tang. Agent- tuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823, 2023

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.