Pith.

Neurons and Cognition · arXiv:2611.17011 v1 · submitted 12 Nov 2026 · 54 references

Random Exploration is Random

Yingkang Luo, Hua-Dong Xiong, Hanbo Xie, Caroline Phelps, Madeline Harms, Ryan Smith, James Waltz, Robert Wilson

The paper says

“Decision making often requires balancing exploitation of familiar options with exploration of uncertain alternatives. One strategy for exploration is random exploration, in which choices become more variable and less tied to the option that currently appears best.”

more of the abstract

However, it remains unclear whether this variability reflects genuine stochasticity or instead a deterministic structure that traditional cognitive models fail to capture. We examined this question using the Horizon Task in an aggregated dataset of 1,392 participants completing over 1,270,695 trials. In this task, exploration becomes more valuable when participants have more future choices, and choice variability increases under longer horizons. We tested whether more expressive neural network models could recover deterministic structure from this additional variability. Although neural networks provided better overall fits to behavior than the traditional model, long-horizon choices remained consistently less predictable than short-horizon choices. These findings show that limited model expressiveness cannot fully account for the increased variability associated with random exploration, suggesting random exploration truly is substantially random.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A careful negative-result study with a stronger title than its evidence: flexible models improve fit but don't close the short- versus long-horizon predictability gap, which is a meaningful result even if 'truly substantially random' overstates it. This line is wrong

3 major objections · 9 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 54 references say.

The paper says, at §2.5, Fig. 3

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.94):

“If similar unmeasured factors influence exploratory choices, their effects would appear as unexplained variability in our models even if they were deterministic.”

Pith says: Objection 1 of 3 · major

The central claim rests on the inability of neural networks to close the H1–H6 predictability gap. However, the authors acknowledge in §3 (b184–b189) that unmeasured factors (e.g., eye movements, fidgeting, internal states) could account for the residual variability. This is a structural limitation of the negative result: the absence of evidence for structure is not evidence of absence.

more of objection 1

The paper's strongest claim (b8) that 'random exploration truly is substantially random' goes beyond what the data can support given this acknowledged limitation. The authors should either soften the conclusion to match the evidence (e.g., 'consistent with a stochastic component') or provide a stronger argument that the measured variables capture the relevant deterministic structure. As written, the conclusion overstates the strength of the negative result.

The paper says, at §2.5, Fig. 3; §2.6, Fig. 4

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The H1–H6 accuracy gap in the subject-level analysis is 6.0 percentage points (b122), and in the first-game analysis it is 7.5 percentage points (b130). The logistic baseline gaps are 6.1 and 11.3 percentage points respectively. While the gap narrows in some comparisons, it is not statistically tested whether the reduction is significant.

more of objection 2

The paper reports confidence intervals for individual accuracies but does not provide a formal test (e.g., interaction between horizon and model type) to assess whether the neural networks significantly reduce the gap relative to the baseline. Without such a test, the claim that 'limited model expressiveness cannot fully account' is weakened because the gap reduction might be within chance. The authors should report a statistical comparison (e.g., bootstrap test of the gap difference) to support their interpretation.

The paper says, at §2.8, Fig. 6

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The within-H6 trial analysis shows that predictability increases as the remaining horizon shortens, which is consistent with the value of future exploration driving variability. However, the final H6 trial accuracy remains below the H1 benchmark (Fig. 6). The authors do not discuss whether this residual gap could be due to the different number of free-choice trials (1 vs. 6) affecting the model's ability to predict, or to other factors like fatigue or attention decrement over longer games.

more of objection 3

A more detailed analysis of the final trial, controlling for the number of preceding choices, would strengthen the interpretation that the gap is specifically due to horizon-related exploration rather than other confounds.

9 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 9

The phrase 'substantially random' is vague. Consider rephrasing to 'consistent with a stochastic component' or 'cannot be fully explained by deterministic structure' to match the paper's own cautious language in the discussion.

The paper says, at §2.3, Table 1

Pith says: minor objection 2 of 9

The first-game fit shows a larger H1–H6 gap (11.3 pp) than the global fit (7.5 pp). The authors note this but do not discuss why the gap is larger in the first game. A brief comment would be helpful.

show 7 more

The paper says, at §2.4, Table 2

Pith says: minor objection 3 of 9

The pseudo-R² values are reported for subject-level models. It would be useful to also report the global model pseudo-R² for comparison, as the global analysis is the simplest test of the hypothesis.

The paper says, at §4.4.1

Pith says: minor objection 4 of 9

The data splitting procedure for subject-level models (80/10/10 within each participant) is described. However, it is not clear whether the test set contains games from participants seen during training (which is the case for subject-level models) or entirely unseen participants (which would be a stricter generalization test). Clarify this distinction.

The paper says, at §4.4.2

Pith says: minor objection 5 of 9

The early stopping patience is five epochs. Given the large dataset, this seems reasonable, but the authors should report the average number of epochs trained to ensure convergence was achieved.

The paper says, at Fig. 3, Fig. 4, Fig. 5

Pith says: minor objection 6 of 9

The error bars are 95% Wilson confidence intervals. It would be helpful to state in the caption whether these are computed across participants or across test trials. For the global model, it is presumably across trials; for subject-level models, it could be across participants. Clarify.

The paper says, at §2.7, Fig. 5

Pith says: minor objection 7 of 9

The cross-game history analysis uses a persistence parameter β that is learned. The authors report low β values, but do not provide confidence intervals or tests against zero. A brief note on whether β is significantly different from zero would strengthen the claim that cross-game history has little effect.

The paper says, at §3, b194–b195

Pith says: minor objection 8 of 9

The second limitation (interpretability) is acknowledged. However, the paper could briefly mention whether any post-hoc analysis of network representations was attempted (e.g., probing for known exploration signals) to partially address this.

The paper says, at References

Pith says: minor objection 9 of 9

Reference [33] (Wang & Wilson, 2026) is listed as published in PLOS Computational Biology. Verify that this is indeed published and not 'in press' or 'to appear', as the paper relies on it for the repeated-game approach.

What would settle it

Re-analyze the data including a proxy for unmeasured internal states, such as trial-by-trial reaction time or response time variability, which is available in most Horizon Task datasets. If including RT as an additional feature in the neural network models reduces the H1–H6 predictability gap by more than 2 percentage points, then the residual variability is partially attributable to a measurable but previously omitted factor, weakening the claim of genuine stochasticity.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic agrees 7 s · 16,827 tokens read, 514 written
  3. Editor wrote the verdict above 21 s · 17,669 tokens read, 1,937 written
  4. Referee major revision: 3 major objections, 9 minor 13 s · 17,140 tokens read, 1,630 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Short version: this is a careful negative-result paper with a stronger title than its evidence. The core finding—that bigger, more flexible models improve overall fit but fail to close the H1–H6 predictability gap—is real and worth knowing. The conclusion that random exploration is "substantially random" is a reasonable inference, not a proof, and the authors mostly say so themselves.

What's new: earlier work used repeated identical games to separate random from deterministic sources; this paper asks whether deep neural networks can recover deterministic structure that simpler models miss. On 1.3 million trials, across MLP, RNN, LSTM, GRU, and transformer-style architectures, the answer is consistently no. The analysis has real strengths: logistic baselines are evaluated out-of-sample on the same splits, subject-level and population-level fits are both reported, first-game analyses rule out learning across games, cross-game history is tested with a learned persistence parameter, and the H6 trajectory analysis shows predictability rising as the remaining horizon shortens. That convergence of evidence is the best part of the paper.

Where it's soft: the inference from absence is structurally limited, and the authors know it. b172 says the residual difference "does not by itself establish irreducible stochasticity," and b184–b189 explicitly list unmeasured deterministic factors—gaze, fidgeting, attention, arousal, boredom—that would look like noise in the current features. So the abstract's "suggesting random exploration truly is substantially random" (b8) overreaches. What the data actually support is: within the measured features, limited model expressiveness cannot close the gap. That is a meaningful constraint on deterministic explanations, but it is not a demonstration that the residual variability is genuinely stochastic. A fully satisfying version of this paper would pair the neural networks with richer measurements or a stronger falsifiable prediction about what additional features would do.

Minor issues: no code or data sharing, a single fixed train/validation/test split, and no explicit significance test on the neural-network accuracy gap across architectures. These are fixable and shouldn't block the core result.

Who this is for: cognitive scientists and neuroscientists working on explore-exploit behavior, and anyone building computational models of choice variability. It deserves a serious referee. I'd send it out, with the request that the authors make code and data available and calibrate the title and abstract to what the design can actually establish.

The strongest objection

The paper's central claim—that random exploration is 'substantially random'—depends on the inability of neural networks to close the predictability gap between H1 and H6. However, this is a negative result: the absence of evidence for structure is not evidence of absence. The authors explicitly acknowledge this limitation (sentence b184: 'A key limitation is that even highly expressive models can only capture deterministic structure that is present in the measured data.') and list unmeasured factors (eye movements, fidgeting, internal states) that could account for the residual variability (b186–b189). The claim therefore rests on an assumption that all relevant deterministic structure is captured by the measured variables (choices, rewards, subject ID, game history). This assumption is not tested and is unlikely to hold given the known influence of physiological and attentional states on choice. The paper's own discussion concedes that future work with richer measurements is needed (b190, b202). Thus, the strongest claim is not supported by the evidence presented; it is a plausible interpretation of a negative result that the authors themselves qualify.

The reader's weakest_assumption correctly identifies that the claim depends on the assumption that the neural networks capture all deterministic structure present in the data. The authors acknowledge this limitation, but it remains structurally inherent to the argument. My concern is essentially the same: the negative result cannot rule out unmeasured confounds. The concrete test I propose (including reaction time) is a feasible check that could strengthen or weaken the claim. The verdict remains CONDITIONAL because the paper's evidence is consistent with the claim but does not definitively establish it. I agree with the reader's assessment and do not recommend changing the verdict.

The referee's report

The paper addresses a fundamental question in decision neuroscience: whether behavioral variability attributed to random exploration is truly stochastic or reflects unmodeled deterministic structure. The study is large-scale (1,392 participants, >1.27M trials) and uses multiple neural network architectures to maximize the chance of capturing any deterministic patterns. The negative result—that even expressive models fail to close the predictability gap—is informative and challenges the view that apparent randomness in exploration can be fully explained by limited model capacity. The authors are transparent about limitations, including the possibility of unmeasured deterministic factors. If the result holds, it supports the notion that random exploration has a genuinely stochastic component, which has implications for theories of decision-making and neural noise.

Pith's reading of the paper (a machine's summary, not the authors')

The paper investigates whether the increased choice variability observed when people have more future opportunities to use new information (long horizon) is genuinely random or merely reflects deterministic structure that simple cognitive models fail to capture. Using neural networks as more expressive models, the authors find that while these networks capture more behavioral structure than traditional logistic models, long-horizon choices remain consistently less predictable than short-horizon choices across multiple analyses. They test several potential sources of deterministic structure—individual differences, learning across games, cross-game history—and find that none eliminate the predictability gap. The paper concludes that limited model expressiveness cannot fully account for the additional variability associated with random exploration, suggesting that random exploration truly has a substantially stochastic component. This matters because it speaks to whether apparent randomness in decision-making reflects irreducible noise or hidden structure, with implications for cognitive models, neuroscience, and AI exploration strategies.

What carries the argument. The key machinery is the comparison of predictive accuracy between short-horizon (H1) and long-horizon (H6) conditions using neural network models of varying expressiveness (MLP, RNN, LSTM, GRU, Transformer, self-attention) as function approximators. The central object is the H1–H6 predictability gap—the difference in how well choices can be predicted in each condition. The argument works by showing that if the gap were due to limited model expressiveness, more expressive models should preferentially improve prediction in H6 and reduce the gap. The persistence of the gap across architectures and analyses carries the argument that the additional variability in H6 is not recoverable deterministic structure.

Core claim. The paper's central claim is that the increased choice variability observed in long-horizon (H6) compared to short-horizon (H1) conditions in the Horizon Task cannot be fully explained by limited model expressiveness, suggesting a genuinely stochastic component of random exploration. This is established by showing that even highly expressive neural network models, which capture more behavioral structure than traditional logistic models, still leave long-horizon choices consistently less predictable than short-horizon choices. The predictability gap persists across multiple analyses: pooled population, subject-level, first-game only, and cross-game history. The paper thus argues that the additional variability associated with random exploration is not merely an artifact of model simplicity but reflects a genuinely stochastic process.

The premise it rests on. The claim that the residual unpredictability is genuinely stochastic depends on the assumption that the neural network models are sufficiently expressive to capture any deterministic structure present in the measured data, but unmeasured deterministic factors (e.g., eye movements, fidgeting, internal states) could account for the gap.

If the paper is right (Pith's inferences, not the authors')

If the central claim is correct, cognitive models of exploration must incorporate a genuinely stochastic component rather than treating apparent randomness as a modeling artifact or hidden deterministic process.

Neuroscientific theories of random exploration should focus on mechanisms that generate irreducible neural noise, such as stochastic neurotransmitter release or noisy decision thresholds, rather than only on deterministic computations.

The findings suggest that behavioral variability in explore-exploit tasks is partly irreducible, which may constrain the precision with which individual choices can be predicted, even with ideal models.

The methodology of using expressive neural networks as a probe for residual stochasticity provides a template for testing whether unexplained variance in other cognitive tasks is genuinely random or reflects unmodeled structure.

The paper's negative result—that the predictability gap persists—does not prove stochasticity but shifts the burden of proof: to demonstrate deterministic structure, future work must measure the specific unobserved variables (e.g., gaze, fidgeting, arousal) that could account for the gap.

The H1–H6 gap decreasing across trials within H6 games suggests that the stochastic component may be strategic: it is engaged when exploration is valuable and reduced when remaining opportunities shrink, consistent with a regulated noise process rather than irreducible neural noise.

If the residual variability is genuinely stochastic, then optimal exploration algorithms in AI might benefit from calibrated noise injection rather than relying solely on deterministic uncertainty-driven exploration, mirroring the human strategy.

The paper's reliance on predictive accuracy as the metric may miss deterministic structure that is not linearly separable or that manifests in choice patterns not captured by next-choice prediction (e.g., sequences of choices across trials).

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Decision making often requires balancing exploitation of familiar options with exploration of uncertain alternatives. One strategy for exploration is random exploration, in which choices become more variable and less tied to the option that currently appears best. However, it remains unclear whether this variability reflects genuine stochasticity or instead a deterministic structure that traditional cognitive models fail to capture. We examined this question using the Horizon Task in an aggregated dataset of 1,392 participants completing over 1,270,695 trials. In this task, exploration becomes more valuable when participants have more future choices, and choice variability increases under longer horizons. We tested whether more expressive neural network models could recover deterministic structure from this additional variability. Although neural networks provided better overall fits to behavior than the traditional model, long-horizon choices remained consistently less predictable than short-horizon choices. These findings show that limited model expressiveness cannot fully account for the increased variability associated with random exploration, suggesting random exploration truly is substantially random.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 1.00

“These findings show that limited model expressiveness cannot fully account for the increased variability associated with random exploration, suggesting random exploration truly is substantially random.”

How the main claim is stated: asserted 0.64

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.42 · human participants 0.98 · controlled comparison 0.22 · randomized assignment 0.03 · states quantitative result 0.85 · compares with prior results 0.89 · mentions code or data availability 0.02

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Decision making often requires balancing exploitation of familiar options with exploration of uncertain alternatives. background 1.00 0.04 0.02 asserted 0.97
One strategy for exploration is random exploration, in which choices become more variable and less tied to the option that currently appears best. background 1.00 0.04 0.02 asserted 0.99
However, it remains unclear whether this variability reflects genuine stochasticity or instead a deterministic structure that traditional cognitive models fail to capture. gap 1.00 0.05 0.03 unclear 0.50
We examined this question using the Horizon Task in an aggregated dataset of 1,392 participants completing over 1,270,695 trials. method 0.94 0.96 0.02 asserted 0.96
In this task, exploration becomes more valuable when participants have more future choices, and choice variability increases under longer horizons. result 0.50 0.50 0.03 asserted 1.00
We tested whether more expressive neural network models could recover deterministic structure from this additional variability. method 0.87 0.92 0.02 hedged possibility 0.49
Although neural networks provided better overall fits to behavior than the traditional model, long-horizon choices remained consistently less predictable than short-horizon choices. result 1.00 0.95 0.12 asserted 1.00
These findings show that limited model expressiveness cannot fully account for the increased variability associated with random exploration, suggesting random exploration truly is substantially random. interpretation 0.99 0.93 0.96 asserted 0.64

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from analysis of existing data 0.88

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 6,121 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (10)
Figure 1 of the paper
Figure 1. The Horizon Task. (A) The first four instructed trials, during which participants are required to select the highlighted green option between the two slot machines. The structure of the forced trials is identical across horizon conditions, differing only in the total number of trials. (B, C) Following the instructed trials, participants complete free-choice trials until the end of the game. During this phase, they choose freely between the two slot machines. In the short-horizon condition (B), participants make one additional free choice, whereas in the long-horizon condition (C),…
Figure 2 of the paper
Figure 2. Choice curves of the Horizon Task. (A, B) Choice curves showing the probability of selecting an option as a function of the difference in observed mean rewards. (A) Probability of choosing the more informative option in the unequal uncertainty condition (i.e., the machine played once during the forced trials). The observed mean reward difference is the reward of the more informative option minus that of the less informative option. (B) Probability of choosing the left option in equal condition as a function of the observed mean reward difference between the left and right options.…
Figure 3 of the paper
Figure 3. Test accuracy by horizon across neural network architectures. (A) Global analysis, where models predicted choices without participant identity. (B) Subject-level analysis, where participant identity was included as an additional input. Each point shows the test accuracy of one architecture, computed separately for H1 and H6 after restoring the checkpoint with the lowest validation loss. Horizontal error bars indicate 95% Wilson confidence intervals. H1 accuracy is shown in blue and H6 accuracy in red. Dashed vertical lines indicate the corresponding horizon-specific test accuracy of…
Figure 4 of the paper
Figure 4. Predictive performance in the first game. Test accuracy was compared between H1 and H6 using only the first game played by each participant. Each point shows the test accuracy of one neural network architecture after restoring the checkpoint with the lowest validation loss, computed sepa￾rately for the H1 and H6 conditions. Horizontal error bars indicate 95% Wilson confidence intervals for test accuracy. H1 accuracy is shown in blue and H6 accuracy is shown in red. Dashed vertical lines indicate the corresponding horizon-specific test accuracy of the first-game logistic regression…
Figure 5 of the paper
Figure 5. Cross-game history analysis. Predictive accuracy was compared between H1 and H6 for recurrent models that incorporated cross-game history through the persistence parameter β. Each point shows the test accuracy of one architecture after restoring the checkpoint with the lowest validation loss, computed separately for the H1 and H6 conditions. Horizontal error bars indicate 95% Wilson confidence intervals for test accuracy. H1 accuracy is shown in blue and H6 accuracy is shown in red. Dashed vertical lines indicate the corresponding horizon-specific test accuracy of the individual-fit…
Figure 6 of the paper
Figure 6. Predictive accuracy across H6 free-choice trials. The red line shows Seq2Seq model accuracy across the six free-choice trials in the H6 condition, from C4 to C9. The dashed blue line indicates the H1 benchmark accuracy. Predictive accuracy increased across later H6 trials as the remaining horizon shortened, but final-trial H6 performance remained below the H1 benchmark. 3 Discussion We investigated whether behavior previously described as random exploration could instead be explained by a deterministic structure that simpler cognitive models fail to capture. Neural network models…
Figure 7 of the paper
Figure 7. MLP Architecture. The model received a single concatenated vector of sequential and static inputs. Sequential inputs consisted of choice–reward pairs (Ct, Rt) from the instructed trials (t ∈ {0, 1, 2, 3}), while static inputs encoded horizon length (H) and uncertainty condition (UC). The architecture consisted of two FC layers (32 and 16 units, respectively) with ReLU activations. The output layer applied a softmax function to generate choice probabilities for the first free-choice trial. C4 denotes the first free-choice response, as trial indexing began at 0. 24
Figure 8 of the paper
Figure 8. RNN-based Architectures. The vanilla RNN and GRU used 64 hidden units, whereas the LSTM used 128 hidden units. The TinyGRU variant had two hidden units. (a) The recurrent layer received only sequential inputs. The final hidden state was concatenated with static inputs. (b) The recurrent layer received both sequential inputs and static inputs at each time step. (a, b) In both architectures, the resulting representation was passed through two FC layers, identical to those used in the MLP architecture, to generate a softmax prediction of the first free-choice response. 25
Figure 9 of the paper
Figure 9. Transformer-based Architectures. Sequential inputs were first projected into a 64-dimensional representation with sinusoidal positional encoding. (a) The Transformer encoder had four attention heads and a feedforward network with 128 hidden units using ReLU activation. (b) The attention￾only variant consisted of eight attention heads. 26
Figure 10 of the paper
Figure 10. Seq2Seq Architecture. The model implemented an encoder–decoder recurrent framework. (a) In the teacher-forcing variant, the decoder received the ground-truth choice at each time step during training to predict the subsequent choice (t ∈ {4, . . . , 9}). (b) In the student-forcing variant, the decoder used its own previous prediction as input to generate the next choice. 27

Figures are the authors', reproduced from the paper.

References (54)
  1. [1] Wilson, R.C., Geana, A., White, J.M., Ludvig, E.A., Cohen, J.D.: Humans use directed and random exploration to solve the explore–exploit dilemma. Journal of Experimental Psychology: General143, 2074–2081 (2014) https://doi.org/10. 1037/a0038199 on Pith
  2. [2] Feng, S.F., Wang, S., Zarnescu, S., Wilson, R.C.: The dynamics of explore–exploit decisions reveal a signal-to-noise mechanism for random exploration. Scientific Reports11(2021) https://doi.org/10.1038/s41598-021-82530-8 on Pithdoi:10.1038/s41598-021-82530-8
  3. [3] Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. IEEE Trans- actions on Neural Networks9(5), 1054–1054 (1998) https://doi.org/10.1109/ TNN.1998.712192 on Pith
  4. [4] Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey (1996) arXiv:cs/9605103 [cs.AI] on PithPith's reviewarXiv
  5. [5] Bellman, R.: A problem in the sequential design of experiments. Sankhy¯ a: The Indian Journal of Statistics (1933-1960)16(3/4), 221–229 (1956) on Pith
  6. [6] Gittins, J.C.: Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society Series B: Statistical Methodology41(2), 148–164 (1979) on Pith
  7. [7] Zhang, S., Yu, A.J.: Forgetful bayes and myopic planning: Human learning and decision-making in a bandit setting. Advances in neural information processing systems26(2013) on Pith
  8. [8] Averbeck, B.B.: Theory of choice in bandit, information sampling and foraging tasks. PLoS computational biology11(3), 1004164 (2015) 19 on Pith
  9. [9] Wilson, R.C., Bonawitz, E., Costa, V.D., Ebitz, R.B.: Balancing exploration and exploitation with information and randomization. Current Opinion in Behavioral Sciences38, 49–56 (2021) https://doi.org/10.1016/J.COBEHA.2020.10.001 on Pithdoi:10.1016/j.cobeha.2020.10.001
  10. [10] Krebs, J.R., Kacelnik, A., Taylor, P.: Test of optimal sampling by foraging great tits. Nature275(5675), 27–31 (1978) on Pith
  11. [11] Watkins, C., et al.: Learning from delayed rewards. (1989) on Pith
  12. [12] Gershman, S.J., Tzovaras, B.G.: Dopaminergic genes are associated with both directed and random exploration. Neuropsychologia120, 97–104 (2018) on Pith
  13. [13] Schulz, E., Gershman, S.J.: The algorithmic architecture of exploration in the human brain. Current opinion in neurobiology55, 7–14 (2019) on Pith
  14. [14] Gershman, S.J.: Deconstructing the human algorithms for exploration. Cognition 173, 34–42 (2018) on Pith
  15. [15] Frank, M.J., Doll, B.B., Oas-Terpstra, J., Moreno, F.: Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nature neuroscience12(8), 1062–1068 (2009) on Pith
  16. [16] Badre, D., Doll, B.B., Long, N.M., Frank, M.J.: Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron73(3), 595– 607 (2012) on Pith
  17. [17] Cavanagh, J.F., Figueroa, C.M., Cohen, M.X., Frank, M.J.: Frontal theta reflects uncertainty and unexpectedness during exploration and exploitation. Cerebral cortex22(11), 2575–2586 (2012) on Pith
  18. [18] Speekenbrink, M., Konstantinidis, E.: Uncertainty and exploration in a restless bandit problem. Topics in cognitive science7(2), 351–367 (2015) on Pith
  19. [19] Daw, N.D., O’doherty, J.P., Dayan, P., Seymour, B., Dolan, R.J.: Cortical substrates for exploratory decisions in humans. Nature441(7095), 876–879 (2006) on Pith
  20. [20] Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science220(4598), 671–680 (1983) https://doi.org/10.1126/science.220.4598.671 on Pithdoi:10.1126/science.220.4598.671
show 34 more
  1. [21] ˇCern´ y, V.: Thermodynamical approach to the traveling salesman problem: An efficient simulation algorithm. Journal of Optimization Theory and Applications 45(1), 41–51 (1985) https://doi.org/10.1007/BF00940812 on Pithdoi:10.1007/bf00940812
  2. [22] Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine IntelligenceP AMI-6(6), 721–741 (1984) https://doi.org/10.1109/ TPAMI.1984.4767596 20 on Pith
  3. [23] Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equation of state calculations by fast computing machines. The Journal of Chemical Physics21(6), 1087–1092 (1953) https://doi.org/10.1063/1.1699114 on Pithdoi:10.1063/1.1699114
  4. [24] Hastings, W.K.: Monte carlo sampling methods using markov chains and their applications. Biometrika57(1), 97–109 (1970) https://doi.org/10.1093/biomet/ 57.1.97 on Pithdoi:10.1093/biomet/
  5. [25] Gelfand, A.E., Smith, A.F.M.: Sampling-based approaches to calculating marginal densities. Journal of the American Statistical Association85(410), 398–409 (1990) https://doi.org/10.1080/01621459.1990.10476213 on Pithdoi:10.1080/01621459.1990.10476213
  6. [26] Faisal, A.A., Selen, L.P., Wolpert, D.M.: Noise in the nervous system. Nature reviews neuroscience9(4), 292–303 (2008) on Pith
  7. [27] Ebitz, R.B., Albarran, E., Moore, T.: Exploration disrupts choice-predictive sig- nals and alters dynamics in prefrontal cortex. Neuron97(2), 450–4619 (2018) https://doi.org/10.1016/j.neuron.2017.12.007 on Pithdoi:10.1016/j.neuron.2017.12.007
  8. [28] Tervo, D.G.R., Proskurin, M., Manakov, M., Kabra, M., Vollmer, A., Branson, K., Karpova, A.Y.: Behavioral variability through stochastic choice and its gating by anterior cingulate cortex. Cell159, 21–32 (2014) https://doi.org/10.1016/j. cell.2014.08.037 on Pithdoi:10.1016/j
  9. [29] Budzillo, A., Duffy, A., Miller, K.E., Fairhall, A.L., Perkel, D.J.: Dopaminer- gic modulation of basal ganglia output through coupled excitation–inhibition. Proceedings of the National Academy of Sciences114(22), 5713–5718 (2017) https://doi.org/10.1073/pnas.1611146114 on Pithdoi:10.1073/pnas.1611146114
  10. [30] Findling, C., Skvortsova, V., Dromnelle, R., Palminteri, S., Wyart, V.: Compu- tational noise in reward-guided learning drives behavioral variability in volatile environments. Nature Neuroscience22(12), 2066–2077 (2019) https://doi.org/10. 1038/s41593-019-0518-9 on Pith
  11. [31] Osborne, L.C., Lisberger, S.G., Bialek, W.: A sensory source for motor variation. Nature437(7057), 412–416 (2005) on Pith
  12. [32] Beck, J.M., Ma, W.J., Pitkow, X., Latham, P.E., Pouget, A.: Not noisy, just wrong: The role of suboptimal inference in behavioral variability. Neuron74, 30–39 (2012) https://doi.org/10.1016/j.neuron.2012.03.016 on Pithdoi:10.1016/j.neuron.2012.03.016
  13. [33] Wang, S., Wilson, R.C.: Separating random and deterministic sources of com- putational noise in explore-exploit decisions. PLOS Computational Biology22, 1014026 (2026) https://doi.org/10.1371/JOURNAL.PCBI.1014026 on Pithdoi:10.1371/journal.pcbi.1014026
  14. [34] Xiong, H.-D., Ji-An, L., Wilson, R.C., Mattar, M.G.: Human strategy adaptation in reinforcement learning resembles policy gradient ascent. bioRxiv (2025) https: 21 //doi.org/10.1101/2025.07.28.667308 on Pithdoi:10.1101/2025.07.28.667308
  15. [35] Gershman, S.J.: Uncertainty and exploration. Decision6(3), 277–286 (2019) https://doi.org/10.1037/dec0000101 on Pithdoi:10.1037/dec0000101
  16. [36] Harms, M.B., Xu, Y., Green, C.S., Woodard, K., Wilson, R., Pollak, S.D.: The structure and development of explore-exploit decision making. Cognitive Psychology150(2024) https://doi.org/10.1016/j.cogpsych.2024.101650 on Pithdoi:10.1016/j.cogpsych.2024.101650
  17. [37] Phelps, C.E., Frisvold, A.E., Sreeram, M., Antoniou, V., Oliveira, L.E., Tooke, S.R., Lu, Y.Z., Lemon, A.C., Fraire, J.U., Delval, A.G., Smith, B.K., Spangenberg, D.H., Mithelavage, M.N., Ngo, M.N., Keller, E.M., Issac, L.J., Harader, S.A., Mizell, J.-M., Wang, S., Keung, W., ... on Pith
  18. [38] Mizell, J.M., Wang, S., Frisvold, A., Alvarado, L., Farrell-Skupny, A., Keung, W., Phelps, C.E., Sundman, M.H., Franchetti, M.K., Chou, Y., Alexander, G.E., Wilson, R.C.: Differential impacts of healthy cognitive aging on directed and random exploration. Psychology and aging39... on Pith
  19. [39] Sadeghiyeh, H., Wang, S., Alberhasky, M.R., Kyllo, H.M., Shenhav, A., Wil- son, R.C.: Temporal discounting correlates with directed exploration but not with random exploration. Scientific Reports10(2020) https://doi.org/10.1038/ s41598-020-60576-4 on Pith
  20. [40] Smith, R., Taylor, S., Wilson, R.C., Chuning, A.E., Persich, M.R., Wang, S., Killgore, W.D.S.: Lower levels of directed exploration and reflective thinking are associated with greater anxiety and depression. Frontiers in Psychiatry12(2022) https://doi.org/10.3389/fpsyt.2021.782136 on Pithdoi:10.3389/fpsyt.2021.782136
  21. [41] Somerville, L.H., Sasse, S.F., Garrad, M.C., Drysdale, A.T., Akar, N.A., Insel, C., Wilson, R.C.: Charting the expansion of strategic exploratory behavior during adolescence. Journal of experimental psychology. General146, 155–164 (2017) https://doi.org/10.1037/XGE0000250 on Pithdoi:10.1037/xge0000250
  22. [42] Waltz, J.A., Wilson, R.C., Albrecht, M.A., Frank, M.J., Gold, J.M.: Differential effects of psychotic illness on directed and random exploration. Computational psychiatry (Cambridge, Mass.)4, 18 (2020) https://doi.org/10.1162/CPSY A 00027 on Pithdoi:10.1162/cpsy
  23. [43] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature521(7553), 436–444 (2015) on Pith
  24. [44] Hornik, K.: Approximation capabilities of multilayer feedforward networks. 22 Neural networks4(2), 251–257 (1991) on Pith
  25. [45] Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math- ematics of Control, Signals and Systems2, 303–314 (1989) https://doi.org/10. 1007/BF02551274 on Pith
  26. [46] Peterson, J.C., Bourgin, D.D., Agrawal, M., Reichman, D., Griffiths, T.L.: Using large-scale experiments and machine learning to discover theories of human decision-making. Science372(6547), 1209–1214 (2021) https://doi.org/10.1126/ science.abe2629 on Pith
  27. [47] Musall, S., Kaufman, M.T., Juavinett, A.L., Gluf, S., Churchland, A.K.: Single- trial neural dynamics are dominated by richly varied movements. Nature Neuro- science22(10), 1677–1686 (2019) https://doi.org/10.1038/s41593-019-0502-4 on Pithdoi:10.1038/s41593-019-0502-4
  28. [48] Li, J.-A., Benna, M.K., Mattar, M.G.: Discovering cognitive strategies with tiny recurrent neural networks. Nature (2025) https://doi.org/10.1038/ s41586-025-09142-4 on Pith
  29. [49] Wang, S., Wilson, R.C., Ebner, N.C., Lighthall, N.R.: Modeling value integra- tion during decision making under uncertainty with the florida and georgia gambling task. Scientific Reports15, 36826 (2025) https://doi.org/10.1038/ s41598-025-08333-3 on Pith
  30. [50] Kingma, D.P., Ba, J.L.: Adam: A method for stochastic optimization. 3rd Inter- national Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings (2014) on Pith
  31. [51] Cho, K., Merri¨ enboer, B.V., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Pro... on Pithdoi:10.3115/v1/d14-1179
  32. [52] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997) on Pith
  33. [53] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems30(2017) on Pith
  34. [54] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neu- ral networks. Advances in Neural Information Processing Systems4, 3104–3112 (2014) 23 Fig. 7MLP Architecture. The model received a single concatenated vector of sequential and static inputs. Sequenti... on Pith

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment