Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Large Language Models Think Too Fast To Explore Effectively

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Traditional LLMs explore poorly because choice signals peak before empowerment is processed, so the model commits early; reasoning models that deliberate longer reach human level.

desk verdict Solid behavioral study of LLM exploration in Little Alchemy 2, but the 'think too fast' causal story rests on SAE layer correlations that don't establish temporal commitment. read the letter →

arxiv 2501.18009 v2 pith:YXJDU4UE submitted 2025-01-29 cs.AI q-bio.NC

Lan Pan , Hanbo Xie , Robert C. Wilson This is my paper · ORCID
classification cs.AIq-bio.NC
keywords LLMexplorationempowermentuncertainty-drivensparseautoencodersLittleAlchemy2reasoningmodelstest-timecomputeopen-endedtask
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models can explore as effectively as humans in an open-ended combinatorial discovery task, Little Alchemy 2, and why they succeed or fail. It finds that most LLMs discover fewer elements than humans within 500 trials, with only the o1 model clearly surpassing humans and DeepSeek-R1 near human level, and that the underperforming models rely almost entirely on uncertainty-driven strategies rather than empowerment. Using per-layer sparse autoencoder probes on LLaMA3.1-70B, it locates uncertainty and choice signals in the first two transformer layers while empowerment is represented only around layer 72. The paper concludes that traditional LLMs 'think too fast': the decision is committed before empowerment information can influence it, which is why they underperform. It positions test-time deliberation, as in reasoning models, as the workable route to better exploration.

What carries the argument

The central object is the layer-wise sparse autoencoder (SAE) probe: for each transformer layer of LLaMA3.1-70B, the paper trains an SAE on embeddings of the trial's available elements, correlates each latent neuron with empowerment, uncertainty, and choice variables, and treats the most-correlated neuron as the locus of that variable's representation. The argument turns on the ordering of these peaks — choice at layer 1, uncertainty at layer 2, empowerment at layer 72 — because it says the model commits to an action before the empowerment signal is computed. Ablating the identified neurons provides the causal component: removing the empowerment neuron suppresses empowerment-guided choices, while removing the early uncertainty neuron degrades task performance catastrophically.

What would settle it

Run LLaMA3.1-70B on the same trials while clamping the layer-72 empowerment neuron to zero and leaving early uncertainty and choice layers intact; if choice probabilities do not shift away from empowerment-guided combinations, the late empowerment representation is not causally involved in the decision and the temporal-mismatch explanation loses its support. A complementary check is time-resolved causal tracing: if injecting empowerment information into layer 1 changes the model's choice, then the decision was not already committed before empowerment was processed.

Watch

Extended reading notes

Core claim

The central claim is that traditional LLMs fail at open-ended exploration because of a temporal mismatch inside the transformer. In LLaMA3.1-70B, the SAE neuron most correlated with uncertainty peaks at layer 2, the neuron most predictive of choices peaks at layer 1, and the neuron correlated with empowerment peaks at layer 72. Because choice and uncertainty are available before empowerment, the paper argues, the model makes premature decisions that favour short-term uncertainty reduction over actions that unlock future possibilities. Regression estimates support this: most models show near-zero empowerment weights, higher temperature increases uncertainty use but not empowerment, and ablating the empowerment neuron selectively reduces empowerment-guided behaviour while ablating the uncertainty neuron disrupts task performance. The paper also reports that prompt engineering, chain-of-thought, and feature steering do not fix the gap, whereas reasoning-trained models that spend more tokens on deliberation reach human-level exploration.

Load-bearing premise

The paper assumes that the layer where a variable shows up most strongly in the model's internal representation is the layer where the model commits to using that variable, so early choice signals mean the decision is made before empowerment is processed.

Editorial extensions

If this is right

  • If the temporal-mismatch account is correct, raising sampling temperature or scaling parameter count will not make traditional LLMs explore well, because the decision is already committed before empowerment is processed.
  • Reasoning models' success suggests that test-time compute scaling and multi-step deliberation are a viable route to human-level exploration in open-ended tasks.
  • The failure of prompt engineering and feature steering implies that fixing exploration requires changing the model's architecture or training, not just its instructions.
  • Ablation results indicate that the early uncertainty layer is load-bearing for basic task competence, while the late empowerment layer specifically modulates empowerment-guided choices, so interventions must target the right layer to change strategy without breaking performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to use the same layer-peak diagnostic on other models and tasks with an explore–exploit tension; if early choice peaks consistently predict poor open-ended exploration, the 'think too fast' mechanism generalizes beyond Little Alchemy 2.
  • Because the paper finds empowerment is represented but unused, the bottleneck may be routing late-layer features to the decision head rather than knowledge; an adapter that reads the layer-72 empowerment signal into the early choice layer could be a direct test.
  • The paper leaves the underlying cause of the temporal mismatch open; comparing models trained with and without explicit reasoning reinforcement would clarify whether the early commitment is architectural or a product of training data and objectives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies open-ended exploration in the game Little Alchemy 2, comparing 29,493 human players with five LLMs: GPT-4o, o1, LLaMA3.1-8B, LLaMA3.1-70B, and DeepSeek-R1. It reports that o1 and DeepSeek-R1 outperform humans while the other models underperform; that most LLMs rely on uncertainty-driven strategies rather than a balance of uncertainty and empowerment; that DeepSeek-R1 produces longer and more diverse reasoning traces than GPT-4o; and, based on SAE analyses of the two LLaMA models, that choice and uncertainty signals peak in early transformer layers while empowerment peaks in later layers. The authors interpret this temporal mismatch as causing premature decision-making and propose that reasoning models avoid this limitation. The appendices report prompt-engineering and activation-intervention experiments.

Significance. If the behavioral findings hold, the paper provides a valuable large-scale benchmark for LLM exploration in an open-ended task, extending prior bandit-based studies to a richer empowerment-driven setting. The combination of high-quality human data, regression-based strategy analysis, reasoning-trace analysis, and SAE-based representational analysis is a strength and gives the paper broad relevance for LLM cognition and AI alignment research. However, the central mechanistic claim — that early-layer SAE correlations show LLMs commit to choices before empowerment is processed — is not currently established by the evidence, and the paper's own ablation results are in tension with the 'too late to matter' interpretation. With that claim revised or supported by further causal analysis, the paper would be a useful contribution.

major comments (4)
  1. [§4.4 and Discussion ('Fast Thinking in Traditional LLMs')] The central causal claim that the early-layer correlation of uncertainty and choice signals, relative to the later empowerment peak, shows the model 'already makes a decision before processing empowerment' is not established by the SAE data. The SAE is trained on per-element embeddings extracted at each layer (Sec. 3.6), so it measures how element representations evolve, not when the model commits to a choice; all transformer layers contribute to the final output, and an easily decodable count-based feature at layer 2 may simply reflect input statistics rather than decision commitment. Moreover, Appendix C.3 reports that ablating the layer-72 empowerment neuron changes the model's empowerment regression weight, which shows that the later empowerment representation is causally usable; this is inconsistent with the 'processed too late to matter' reading and equally consistent with 'represented but weighted weakly.' The sentence 'This temporal mismatch causes premature decision-making' should be substantially softened or supported with a method that directly traces when decisions are causally determined.
  2. [Abstract vs. §4.1] The abstract states that 'most LLMs underperform compared to humans, except for the o1 model,' but Section 4.1 reports that DeepSeek-R1 also significantly outperformed humans (t = 3.40, p < 0.027). This is a factual discrepancy in the headline result. The abstract and any summarizing statements should include DeepSeek-R1 as a model that exceeded human performance; the current wording excludes a model that the paper's own analysis finds above human level.
  3. [§4.4, Figure 5, Appendix D] The SAE correlations are reported without confidence intervals, a null distribution, or correction for selecting the maximum over 8,192 (or 4,096) latent neurons per layer. Without a permutation baseline, the reported r = 0.55 at layer 72 cannot be distinguished from chance-level maxima under selection. In addition, the SAE analysis is run only on LLaMA3.1-70B and LLaMA3.1-8B, so the Discussion's implication that reasoning models avoid the early/late temporal mismatch is not directly tested in o1 or DeepSeek-R1. Please report the full distribution of correlations across neurons, a noise baseline, and either SAE results for a reasoning model or an explicit limitation.
  4. [§3.3.1] The empowerment update factors increase_factor and decrease_factor are never specified. Because these factors update the empowerment regressors trial by trial, the regression coefficients reported in Section 4.2 and the strategy comparison in Figure 3 depend on unstated parameters. Reporting the values used (or a sensitivity analysis) is necessary for reproducibility and for comparing across models and humans.
minor comments (5)
  1. [§4.1] There is a typo: 'o1 discouvered' should be 'o1 discovered.'
  2. [§3.5] The accuracy of GPT-4.1 as an automated labeler is not validated; please report agreement with human annotations on a sample of reasoning sentences.
  3. [§4.3] The phrase 'from in the subset' is ungrammatical, and the text should state explicitly whether the same first 150 trials are analyzed for both DeepSeek-R1 and GPT-4o.
  4. [Appendix E and §4.4] The claim that the SAE pattern is 'replicated' in LLaMA3.1-8B needs clarification: for LLaMA3.1-8B the empowerment peak is at layer 22 and choice peak at layer 15, whereas for LLaMA3.1-70B the corresponding peaks are layers 72 and 1; the text should explain what counts as a replication given the different model depths.
  5. [§4.2] The random-sampling procedure used to balance chosen and not-chosen elements is underspecified; please state how many negative examples per trial were sampled and whether the sampling was repeated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core regression and SAE analyses use external variables and falsifiable interventions.

full rationale

The paper's central claims are not circular by construction. The empowerment values used in the GLMM regressions are borrowed from an external neural network model (Brändle et al. [5]) and are not fitted to LLM behavior; uncertainty is defined from trial counts. The behavioral regressions are descriptive, not predictions derived from first principles. The SAE analysis identifies neurons by correlation with externally defined variables and then tests their causal role through ablation, which is a falsifiable intervention rather than a tautology. The temporal-mismatch conclusion ('uncertainty and choices processed earlier than empowerment') is an interpretive leap from layer-wise correlations, not an equation that reduces to its inputs. Concerns about whether layer of peak correlation reflects decision commitment are validity/identifiability issues, not circularity. Self-citations appear only in related-work contexts and do not carry the main argument. No fitted parameter is renamed as a prediction, and no load-bearing premise is justified solely by self-citation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The main external inputs are the empowerment model from prior work, the uncertainty formula, and the dynamic update factors. No new physical or theoretical entities are introduced.

free parameters (2)
  • empowerment dynamic update factors = not specified
    Section 3.3.1 defines L(E(e_i)) with increase_factor and decrease_factor, but never gives their values. These factors update the empowerment values used in all strategy regressions.
  • SAE sparsity regularization strength = 1e-6
    Appendix D sets the L2 regularization to 1e-6; this is a hyperparameter, and the paper does not show that the layer-localization results are robust to it.
assumptions (3)
  • domain assumption The empowerment neural network from Brändle et al. provides valid empowerment values for all combinations and elements in Little Alchemy 2.
    Section 3.3.1: 'we use the same empowerment value of each combination from the neural network model'; the regression and SAE analyses depend on this external model's accuracy.
  • domain assumption SAE neuron correlation with a variable implies that the model represents and uses that variable in that layer, and ablation of that neuron isolates its causal role.
    Section 3.6 and Figure 5; the causal claim about 'thinking too fast' depends on treating layer-wise correlation peaks as temporal processing order.
  • domain assumption The 500-trial LLM setting is comparable to the human benchmark, despite humans choosing when to stop.
    Section 4.1 compares LLMs at 500 trials to human averages; humans may stop earlier, which could affect strategy and performance comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models Think Too Fast To Explore Effectively." pith.science (2026). https://pith.science/paper/YXJDU4UE

@misc{pith2026250118009,
  author       = {Pith},
  title        = {Pith review of: Large Language Models Think Too Fast To Explore Effectively},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YXJDU4UE}},
  note         = {Machine review of arXiv:2501.18009}
}
read the original abstract

Large Language Models (LLMs) have emerged with many intellectual capacities. While numerous benchmarks assess their intelligence, limited attention has been given to their ability to explore--an essential capacity for discovering new information and adapting to novel environments in both natural and artificial systems. The extent to which LLMs can effectively explore, particularly in open-ended tasks, remains unclear. This study investigates whether LLMs can surpass humans in exploration during an open-ended task, using Little Alchemy 2 as a paradigm, where agents combine elements to discover new ones. Results show most LLMs underperform compared to humans, except for the o1 model, with traditional LLMs relying primarily on uncertainty-driven strategies, unlike humans who balance uncertainty and empowerment. Results indicate that traditional reasoning-focused LLMs, such as GPT-4o, exhibit a significantly faster and less detailed reasoning process, limiting their exploratory performance. In contrast, the DeepSeek reasoning model demonstrates prolonged, iterative thought processes marked by repetitive analysis of combinations and past trials, reflecting a more thorough and human-like exploration strategy. Representational analysis of the models with Sparse Autoencoders (SAE) revealed that uncertainty and choices are represented at earlier transformer blocks, while empowerment values are processed later, causing LLMs to think too fast and make premature decisions, hindering effective exploration. These findings shed light on the limitations of LLM exploration and suggest directions for improving their adaptability.

Figures

Figures reproduced from arXiv: 2501.18009 by the authors.

Figure 1
Figure 1. A: LLMs Game Process. LLMs select two elements per trial based on the inventory and trial history. B: Human Game Interface. Players select two elements to discover new elements, added to the inventory. C: LLMs and Human Performance. 4.1 Most LLMs Performed Worse Than Humans, Except o1 From 29,493 human players, 90% completed fewer than 500 trials. Experiments were set up with 500 trials for the LLMs. On average, LLa… view at source ↗
Figure 2
Figure 2. Human and LLMs different Temperatures’ Performance. LLM and Human Perfor￾mance Across Temperatures. For LLMs, we set four temperatures (0, 0.3, 0.7, 1). LLMs (GPT-4o, LLaMA3.1-8B, LLaMA3.1-70B) achieve their best performance at temperature = 1. 4.2 LLMs Primarily Use Uncertainty-driven Strategies but Not Empowerment To examine the exact strategies that the models are using, we calculated uncertainty and empowerment … view at source ↗
Figure 3
Figure 3. Regression Estimates by Temperature and Model. All models show lower empowerment weights than humans, except o1. As temperature increases, uncertainty weights rise, with o1 showing the highest weights across all models and humans. 4.3 Reasoning Depth and Token Usage in DeepSeek-R1 and GPT-4o To further elucidate the differences in reasoning depth between DeepSeek-R1 and GPT-4o, we analyzed their reasoning processes … view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Comparison of reasoning depth and token usage between DeepSeek-R1 and GPT-4o. A: Per-Trial Reasoning Depth. DeepSeek-R1 shows substantially longer reasoning sequences and consistently employs all reasoning labels, while GPT-4o exhibits significantly shorter sequences a…
Figure 5
Figure 5. Figure 5: SAE Correlation Analysis. Maximum correlation of uncertainty values across layers, peaking at layer 2. Maximum correlation of empowerment values across layers, peaking at layer 72. Maximum beta weight of choices across layers, peaking at layer 1. 5 Discussion and Concl…
Figure 6
Figure 6. Figure 6: Game Difficulty vs. Inventory Size. Based on the real game tree, each inventory size has a different success probability. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Behavioral Categories of LLMs at Different Temperatures. Each trial is categorized into five conditions: (1) Failure with Existing Combination: The trial repeats a previous combination that does not generate a new element. (2) Failure with New Combination: The trial us…
Figure 8
Figure 8. Figure 8: A: LLMs Game Original Prompt. The prompt for each trial consists of three parts: the system prompt, which provides the game rule guide; the current inventory including those from the beginning and discoveries during the game; and the trial history. B: LLMs Game Prompt …
Figure 9
Figure 9. Figure 9: A: Best Temperature of Each Model and Human Performance. B: Best Temperature of Each Model and Human Behaviors. Choose the LLM models’ (GPT-4o, LLaMA3.1-8B, LLaMA3.1-70B) best performance at temperature = 1, and compare it with human and o1, GPT-4o prompt-engineering (…
Figure 10
Figure 10. Figure 10: A: Human and LLMs Best Temperatures’ Behaviors. According to whether the combination selected by each trial is repeated, successful, and initial, the behavior of each LLM trial is divided into 5 categories. Compare the temperature at which LLM performs best with human…
Figure 11
Figure 11. Figure 11: The reasoning process of DeepSeek-R1 in Trial 200. The model explores possible combinations (blue color part) for discovering a new element in the game by systematically reviewing inventory elements (orange color part), prior attempts (blue color part), and logical in…
Figure 12
Figure 12. Figure 12: Comparison of Transition Probabilities between DeepSeek-R1 and GPT-4o. Transition matrices indicate that DeepSeek engages extensively in iterative cycles of combination analysis and past trial analysis, reflecting a detailed exploration strategy. Conversely, GPT-4o ra…
Figure 13
Figure 13. Figure 13: A: LLaMA3.1-70B Average Inventory of Uncertainty Intervention. Set 5 different levels of uncertainty intervention (0.0, 0.5, 0.7, 1.0, 2.0). Increasing the uncertainty intervention progressively disrupts the model’s ability to complete the task, indicating the critica…
Figure 14
Figure 14. Figure 14: LLaMA3.1-70B Intervention Regression Results. The regression estimates for empower￾ment, and uncertainty under the original condition (LLaMA3.1-70B, temperature = 1), empowerment intervention (set to 0), and uncertainty intervention (set to 0). 19 [PITH_FULL_IMAGE:fi…
Figure 15
Figure 15. Figure 15: Sparse Autoencoder (SAE) Training Metrics. Each row represents different model architectures. From left to right, the panels illustrate the layer-wise test L2 norm, test reconstruction loss, and the number of active neurons during training. The top row corresponds to …
Figure 16
Figure 16. Figure 16: A and Figure 16B, show that setting the empowerment layer to zero had a minimal effect on regression estimates and only slightly reduced model performance, suggesting that the empowerment layer has a limited role in sustaining task performance. In contrast, setting th…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Auditable Agent Platform For Automated Molecular Optimisation

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A hierarchical multi-agent LLM platform with recorded provenance improved average predicted binding affinity for AKT1 by 31%, while single-agent runs favored drug-likeness.

Reference graph

Works this paper leans on

41 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning

    Prithviraj Ammanabrolu and Mark O Riedl. Playing text-adventure games with graph-based deep reinforcement learning. arXiv preprint arXiv:1812.01628, 2018

    work page Pith review arXiv 2018

  2. [2]

    How to avoid being eaten by a grue: Structured exploration strategies for textual worlds

    Prithviraj Ammanabrolu, Ethan Tien, Matthew Hausknecht, and Mark O Riedl. How to avoid being eaten by a grue: Structured exploration strategies for textual worlds. arXiv preprint arXiv:2006.07409, 2020

    arXiv 2006

  3. [3]

    Using cognitive psychology to understand gpt-3

    Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120, 2023

    work page 2023

  4. [4]

    R-max-a general polynomial time algorithm for near-optimal reinforcement learning

    Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3(Oct):213–231, 2002

    work page 2002

  5. [5]

    Empowerment contributes to exploration behaviour in a creative video game

    Franziska Brändle, Lena J Stocks, Joshua B Tenenbaum, Samuel J Gershman, and Eric Schulz. Empowerment contributes to exploration behaviour in a creative video game. Nature Human Behaviour, 7(9):1481–1489, 2023

    work page 2023

  6. [6]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and ...

    2023

  7. [7]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018

    arXiv 2018

  8. [8]

    Cortical substrates for exploratory decisions in humans

    Nathaniel D Daw, John P O’doherty, Peter Dayan, Ben Seymour, and Raymond J Dolan. Cortical substrates for exploratory decisions in humans. Nature, 441(7095):876–879, 2006

    work page 2006

Show all 41 references
  1. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI Team. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  2. [10]

    Sparse autoencoders reveal temporal difference learning in large language models

    Can Demircan, Tankred Saanum, Akshay K Jagadish, Marcel Binz, and Eric Schulz. Sparse autoencoders reveal temporal difference learning in large language models. arXiv preprint arXiv:2410.01280, 2024

    2024 arXiv

  3. [11]

    First return, then explore

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580–586, 2021

    2021

  4. [12]

    Trait somatic anxiety is associated with reduced directed exploration and underestimation of uncertainty.Nature Human Behaviour, 7(1):102–113, 2023

    Haoxue Fan, Samuel J Gershman, and Elizabeth A Phelps. Trait somatic anxiety is associated with reduced directed exploration and underestimation of uncertainty.Nature Human Behaviour, 7(1):102–113, 2023

    2023

  5. [13]

    Minedojo: Building open-ended embodied agents with internet-scale knowledge

    Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:18343–18...

    2022

  6. [14]

    Llama rider: Spurring large language models to explore the open world

    Yicheng Feng, Yuxuan Wang, Jiazheng Liu, Sipeng Zheng, and Zongqing Lu. Llama rider: Spurring large language models to explore the open world. arXiv preprint arXiv:2310.08922, 2023

    2023 arXiv

  7. [15]

    Deconstructing the human algorithms for exploration

    Samuel J Gershman. Deconstructing the human algorithms for exploration. Cognition, 173:34– 42, 2018. 10

    2018

  8. [16]

    Reinforcement learning: A survey

    Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237–285, 1996

    1996

  9. [17]

    Can large language models explore in-context? arXiv preprint arXiv:2403.15371, 2024

    Akshay Krishnamurthy, Keegan Harris, Dylan J Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? arXiv preprint arXiv:2403.15371, 2024

    2024 arXiv

  10. [18]

    The llama 3 herd of models

    Llama Team. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  11. [19]

    Sparse autoencoder

    Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1–19, 2011

    2011

  12. [20]

    Evolve: Evaluating and optimizing llms for exploration

    Allen Nie, Yi Su, Bo Chang, Jonathan N Lee, Ed H Chi, Quoc V Le, and Minmin Chen. Evolve: Evaluating and optimizing llms for exploration. arXiv preprint arXiv:2410.06238, 2024

    2024 arXiv

  13. [21]

    Gpt-4o system card

    OpenAI Team. Gpt-4o system card. https://openai.com/index/gpt-4o-system-card/ , 2024

    2024

  14. [22]

    Openai o1 system card

    OpenAI Team. Openai o1 system card. https://openai.com/index/ openai-o1-system-card/ , 2024

    2024

  15. [23]

    Deep exploration via bootstrapped dqn

    Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016

    2016

  16. [24]

    (more) efficient reinforcement learning via posterior sampling

    Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013

    2013

  17. [25]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778–

  18. [26]

    Planning to explore via self-supervised world models

    Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International conference on machine learning, pages 8583–8592. PMLR, 2020

    2020

  19. [27]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  20. [28]

    Uncertainty and exploration in a restless bandit problem

    Maarten Speekenbrink and Emmanouil Konstantinidis. Uncertainty and exploration in a restless bandit problem. Topics in cognitive science, 7(2):351–367, 2015

    2015

  21. [29]

    Testing theory of mind in large language models and humans

    James W A Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, et al. Testing theory of mind in large language models and humans. Nature Human Behaviour, 8(7):1285–1295, 2024

    2024

  22. [30]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  23. [31]

    Gpt-4 emulates average-human emotional cognition from a third-person perspective

    Ala N Tak and Jonathan Gratch. Gpt-4 emulates average-human emotional cognition from a third-person perspective. arXiv preprint arXiv:2408.13718, 2024

    2024 arXiv

  24. [32]

    V oyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  25. [33]

    Emergent analogical reasoning in large language models

    Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526–1541, 2023

    2023

  26. [34]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  27. [35]

    Deep exploration as a unifying account of explore-exploit behavior

    Robert Wilson, Siyu Wang, Hashem Sadeghiyeh, and Jonathan D Cohen. Deep exploration as a unifying account of explore-exploit behavior. 2020. 11

    2020

  28. [36]

    Balancing exploration and exploitation with information and randomization

    Robert C Wilson, Elizabeth Bonawitz, Vincent D Costa, and R Becket Ebitz. Balancing exploration and exploitation with information and randomization. Current opinion in behavioral sciences, 38:49–56, 2021

    2021

  29. [37]

    Humans use directed and random exploration to solve the explore–exploit dilemma

    Robert C Wilson, Andra Geana, John M White, Elliot A Ludvig, and Jonathan D Cohen. Humans use directed and random exploration to solve the explore–exploit dilemma. Journal of experimental psychology: General, 143(6):2074, 2014

    2014

  30. [38]

    Step back to leap forward: Self-backtracking for boosting reasoning of language models

    Xiao-Wen Yang, Xuan-Yi Zhu, Wen-Da Wei, Ding-Chu Zhang, Jie-Jing Shao, Zhi Zhou, Lan- Zhe Guo, and Yu-Feng Li. Step back to leap forward: Self-backtracking for boosting reasoning of language models. arXiv preprint arXiv:2502.04404, 2025

    2025 arXiv

  31. [39]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  32. [40]

    Counting to explore and generalize in text-based games

    Xingdi Yuan, Marc-Alexandre Côté, Alessandro Sordoni, Romain Laroche, Remi Tachet des Combes, Matthew Hausknecht, and Adam Trischler. Counting to explore and generalize in text-based games. arXiv preprint arXiv:1806.11525, 2018

    2018 arXiv

  33. [41]

    empowerment

    Wojciech K Zajkowski, Malgorzata Kossut, and Robert C Wilson. A causal role for right frontopolar cortex in directed, but not random, exploration. Elife, 6:e27430, 2017. 12 A The Game Difficulty We use a real game tree to calculate the probability of each player succeeding as ...

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.