REVIEW 8 cited by
Can large language models explore in-context?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We investigate the extent to which contemporary Large Language Models (LLMs) can engage in exploration, a core capability in reinforcement learning and decision making. We focus on native performance of existing LLMs, without training interventions. We deploy LLMs as agents in simple multi-armed bandit environments, specifying the environment description and interaction history entirely in-context, i.e., within the LLM prompt. We experiment with GPT-3.5, GPT-4, and Llama2, using a variety of prompt designs, and find that the models do not robustly engage in exploration without substantial interventions: i) Across all of our experiments, only one configuration resulted in satisfactory exploratory behavior: GPT-4 with chain-of-thought reasoning and an externally summarized interaction history, presented as sufficient statistics; ii) All other configurations did not result in robust exploratory behavior, including those with chain-of-thought reasoning but unsummarized history. Although these findings can be interpreted positively, they suggest that external summarization -- which may not be possible in more complex settings -- is important for obtaining desirable behavior from LLM agents. We conclude that non-trivial algorithmic interventions, such as fine-tuning or dataset curation, may be required to empower LLM-based decision making agents in complex settings.
Forward citations
Cited by 8 Pith papers
-
ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interaction
ABBEL trains LLM agents to act from a compact natural-language belief state; belief-quality and brevity rewards close most of the gap with full-context agents.
-
Behavioral Exploration: Learning to Explore via In-Context Adaptation
A coverage-conditioned behavioral cloning policy adapts in-context to its own history, making robots explore new expert-like behaviors online without online reinforcement learning.
-
e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs
e3 couples asymmetric skill chaining, negative-gradient RL, and a difficulty/budget curriculum so a 1.7B model extrapolates test-time compute to 2x its training budget and sets reported <2B state-of-the-art on AIME/HMMT 2025.
-
Large Language Models Think Too Fast To Explore Effectively
In Little Alchemy 2, most LLMs discover fewer elements than humans and rely on uncertainty rather than empowerment; reasoning models o1 and DeepSeek-R1 explore more effectively.
-
Disentangling Exploration of Large Language Models by Optimal Exploitation
Exploration by LLM agents can be measured separately from exploitation using an optimal exploitation oracle, and most models explore poorly, with exploration performance correlated to reasoning ability.
-
Large Language Model-Enhanced Multi-Armed Bandits
Using an LLM as a reward predictor inside Thompson sampling and regression-oracle bandits outperforms LLM direct arm selection in the tested tasks.
-
Meta-Prompt Optimization for LLM-Based Sequential Decision Making
EXPO uses adversarial bandit weighting over LLM-generated prompt variations to optimize the meta-prompt of LLM-based sequential decision-making agents, improving performance on optimization and bandit tasks.
-
A non-ergodic framework for understanding emergent capabilities in Large Language Models
Claims that LLMs are non-ergodic and that capability emergence obeys a resource-constrained 'adjacent possible' equation, but the derivation is an analogy and the experiments are too small to validate it.
Discussion (0). Continue with ORCID to comment.