Pith. sign in

REVIEW 4 major objections 4 minor 50 references

An RL agent can learn which planning horizon each task needs, without manual tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:42 UTC pith:OBUL5RJU

load-bearing objection An honest, plausible algorithm for adaptively combining horizon-specific critics, but the headline claim that it 'identifies effective discount factors' is not backed by the theory and only partly backed by the experiments; worth a serious look with revision. the 4 major comments →

arxiv 2607.20656 v3 pith:OBUL5RJU submitted 2026-07-22 cs.LG cs.AI

Adaptive Multi-Horizon Reinforcement Learning

Manoosh Samiei , Doina Precup , Paul Masset This is my paper
classification cs.LG cs.AI
keywords reinforcement learningmulti-timescalediscount factormixture of expertscontinual learningBellman residualtemporal abstractiongating network
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to show that a reinforcement-learning agent does not need to commit to a single discount factor, which fixes one temporal horizon. Instead, it can maintain several value functions, each with a different horizon, and learn state-dependent weights that mix them into one action-value function. The authors claim this mixture adaptively identifies the most effective horizon for each task, so the agent can handle task switches that demand short- versus long-horizon planning without manual discount-factor tuning. If true, this would give a simple, parameter-efficient mechanism for continual reinforcement learning and align with biological evidence that dopamine circuits encode multiple timescales. The supporting evidence is a set of MiniGrid experiments in which the agent keeps returns near the best single-discount baselines across three sequential tasks.

Core claim

The paper's central claim is that minimizing the undiscounted Bellman error over a state-dependent mixture of horizon-specific critics recovers the discount factor that maximizes undiscounted episodic return. Concretely, the agent learns several Q-functions, each trained with a different discount factor, and a gating network produces softmax weights that combine them into Q_mix. Q_mix is updated with an undiscounted (gamma = 1) Expected SARSA objective, which the authors show, under stated assumptions, concentrates weight on the critic with the smallest undiscounted Bellman residual. Empirically, in a continual setting with three sequentially switched MiniGrid tasks, the mixture keeps episod

What carries the argument

The central mechanism is a learnable gating network over K horizon-dependent Q-critics. The gating computes logits z = W x + b, normalizes them with a softmax to obtain weights w_i(s), and forms Q_mix(s,a) = sum_i w_i(s) Q_i(s,a). The gating parameters are updated by gradient descent on the squared undiscounted TD error, yielding updates W <- W + alpha * delta * [w ⊙ (Q - Q_mix)] x^T and b <- b + alpha * delta * [w ⊙ (Q - Q_mix)]. This semi-gradient rule is what lets the mixture move toward the convex combination of critics that best satisfies the undiscounted Bellman equation, which the paper argues is equivalent to selecting the best temporal horizon.

Load-bearing premise

The entire argument rests on the assumption that the planning horizon which maximizes total collected reward is also the one whose value estimates best satisfy the undiscounted reward-prediction equation; the paper does not justify this relationship, and its own Four Rooms results suggest it can fail in some seeds.

What would settle it

Construct an MDP where the discount factor with the highest undiscounted episodic return has a larger undiscounted Bellman residual than another discount factor. According to Theorem 1, the mixer would then concentrate weight on the suboptimal horizon. The paper's own Figure 10 already provides a candidate test: for the five Four Rooms seeds that assign higher weight to suboptimal discount factors, check whether those discount factors also have smaller undiscounted Bellman residuals than the empirically optimal 0.99219; if they do not, Assumption 3 is violated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agents can adapt their effective planning horizon to a new task without any manual discount-factor tuning, which is directly useful for continual learning.
  • The learned mixture weights provide a per-state readout of which temporal horizon is currently most relevant, potentially enabling agents to switch between short- and long-horizon reasoning as the environment changes.
  • In the experiments, the mixture matches the best single-discount baselines on episodic return, and in the Four Rooms task it reliably collects the time-constrained jackpot even when a standard SARSA agent with the same discount factor fails.
  • The method is demonstrated with tabular Expected SARSA(λ), but the paper frames it as extensible to neural and transformer-based value function approximators.
  • The theoretical result implies a new criterion for choosing among multiple critics or value representations: pick the one with the smallest undiscounted Bellman residual.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the core claim holds, the gating weights could be interpreted as a learned, state-dependent measure of how far into the future the current situation demands planning, offering a bridge to neuroscience models of flexible temporal discounting.
  • Because a sum of exponentially discounted value functions can approximate hyperbolic discounting, this state-dependent mixture may provide a mechanism for adaptive non-exponential discounting; one testable extension is to compare learned weight dynamics to recordings from dopamine neurons tuned to different reward time horizons.
  • The paper's theoretical argument assumes the best-horizon critic also has the smallest Bellman residual; if that assumption fails, the method might still work through interpolation rather than selection. A diagnostic experiment would correlate, per state, the gating weights with the per-critic undiscounted Bellman residuals.
  • The same mixture-of-critics architecture could be applied beyond discount factors, e.g., mixing critics with different eligibility traces, reward models, or representation granularities, to adapt other algorithmic hyperparameters.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a multi-horizon reinforcement learning method in which several SARSA(λ) critics, each using a different discount factor, are combined by a state-dependent softmax gating network. The mixture is trained with an undiscounted expected-SARSA update, and the gating parameters are updated by semi-gradient descent on the squared TD error. The authors first present motivating experiments on MiniGrid showing that the best discount factor depends on reward reachability and frequency, then evaluate the method in a continual setting with three sequential tasks (foraging, goal reaching, four rooms). They claim the mixture identifies effective discount factors and achieves near-optimal performance across task switches.

Significance. If the central claims are supported, the method would be a useful alternative to manual discount-factor tuning and to single-horizon meta-gradient methods, particularly in continual learning settings. The paper has several strengths: the motivating experiments in Section 3 are carefully controlled and directly demonstrate the dependence of optimal discounting on reward structure; the gradient derivation in the Appendix is explicit and correct as a semi-gradient update; and the paper candidly reports seed-dependent variability and limitations. However, the load-bearing theoretical and empirical claims about horizon selection are not established. The main theoretical result is conditional on an assumption that is essentially the conclusion it seeks to prove, and the paper's own Figure 10 shows that in the Four Rooms task five of ten seeds converge to suboptimal discount factors. The absence of baseline learning curves further weakens the empirical contribution. The framework is promising, but the current manuscript does not justify the abstract's unqualified claim that the method identifies effective discount factors.

major comments (4)
  1. [Appendix, Theorem 1] Assumption 3 (Eq. 33) states that the critic associated with the optimal horizon has the smallest undiscounted Bellman residual. This is precisely the relationship the theorem needs to establish before it can conclude that minimizing the Bellman residual 'recovers the temporal horizon that maximizes the undiscounted episodic return.' As written, the theorem is conditional on its conclusion: it proves only that if the optimal horizon is the one with smallest residual, the mixture weights concentrate on that horizon. This does not support the paper's headline claim that the method 'identifies effective discount factors.' The authors should either remove that claim or provide independent evidence for Assumption 3; the current Figure 10 evidence appears contrary in a substantial fraction of runs.
  2. [Section 4.1, Figure 10] The abstract and Section 4.1 claim the approach identifies effective discount factors. In the Four Rooms task, five of ten seeds assign higher weight to discount factors that Figure 8b identifies as suboptimal. The later sentence in Section 4.1 — 'the learned mixture captures task-relevant structure beyond simply identifying an optimal discount factor' — implicitly concedes that the selection mechanism is not doing what the headline claims. The paper should report a quantitative selection accuracy (fraction of seeds whose highest-weight discount factor is among the best-performing set) and a statistical comparison. If selection is unreliable, the claims must be qualified.
  3. [Section 4.1, Figure 9] The claim that the method achieves 'near-optimal performance' and is 'comparable to the best individual discount-factor baselines' is not directly supported by any baseline curves. Figure 9 shows only the proposed method. To support the central empirical claim, the authors should plot or tabulate, for the same seeds and setting, the performance of the oracle best fixed discount factor, a representative adaptive method such as meta-gradient (Hessel et al., 2019), and a fixed-uniform or random mixture. Without such comparisons, the reader cannot assess whether the multi-horizon mixture improves over tuning gamma per task or over a fixed mixture.
  4. [Appendix, Theorem 1, Assumption 4] Assumption 4 ('Bellman residuals from different experts do not cancel in the convex combination') is stated without verification and is not a natural consequence of the preceding assumptions. In a convex combination, the squared residual of the mixture can be smaller than each individual residual precisely because of cancellation between positive and negative residuals. If cancellation occurs, minimizing the mixture residual will not necessarily concentrate weight on the expert with the smallest individual residual. This assumption is load-bearing for the conclusion w_γ*→1. The authors should either test it empirically (e.g., by measuring residual correlations) or weaken the theorem to a statement about sufficient conditions, and should not use the theorem as support for the practical claim without such qualification.
minor comments (4)
  1. [Section 4, Eq. (1) and Appendix] The derivation correctly identifies the update as a semi-gradient, but it would be useful to state explicitly that the target y in Eq. (6) depends on the current gating parameters through Q_mix(s', a'), and that this dependence is ignored in the update. This is standard for semi-gradient methods, but the explicit statement would avoid confusion about the exact objective being optimized.
  2. [Sections 3.3 and 4] Section 3.3 says the agent receives 'full grid observations,' while Section 4 defines the state representation x=φ(s) as the agent's (x,y) position and orientation. This means the gating network does not see the locations of rewards, lava, or goals. The discrepancy should be clarified, and the limitation of the state representation for the claimed 'state-dependent' weighting should be discussed.
  3. [Section 4.1, Figure 9] The paper states that 'the episode return remains between 36 and 40 across all runs,' but Figure 9a appears to show substantial variance and some runs below 36, especially around task switches. Reporting mean and standard error over seeds, or individual seed curves with a legend, would make the claim more precise.
  4. [References] The paper would benefit from comparing with the adaptive-discounting baselines it cites, such as meta-gradient methods (Hessel et al., 2019) and multi-horizon approaches (Fedus et al., 2019). The related-work section mentions these but the experiments do not include them; at minimum, a discussion of why they were not included would strengthen the positioning.

Circularity Check

1 steps flagged

Theorem 1's horizon-selection conclusion restates its own Assumption 3; the headline 'identifies effective discount factors' is assumed, not derived, and Fig. 10 contradicts it in 5/10 seeds.

specific steps
  1. self definitional [Appendix, Exploratory Theoretical Analysis, Theorem 1, Assumption 3 (Eq. 33) and conclusion (Eq. 36)]
    "3. The critic associated with the optimal horizon provides the most accurate approximation to the undiscounted Bellman equation: E[δ²γ*] < E[δ²γi], ∀i≠* ... Therefore, minimizing the undiscounted Bellman error recovers the temporal horizon that maximizes the undiscounted episodic return: γ* = arg max γi J(πγi)."

    Theorem 1 defines γ* (Eq. 31) as the horizon maximizing undiscounted episodic return, then Assumption 3 stipulates that this same γ* has the smallest undiscounted Bellman residual. The theorem's concluding Eq. 36 simply restates that stipulation: minimizing the residual 'recovers' γ*. The weight-concentration step (wγ*→1) is a convex-minimization consequence, but the horizon-identification half of the claim is exactly the assumption, not a derivation. Because the Abstract and Section 4.1 use this as the basis for the claim that the method 'identifies effective discount factors,' that headline claim is assumed rather than proven. Fig. 10 further shows 5/10 Four Rooms seeds assigning more weight to suboptimal discount factors, so the assumption is not empirically established.

full rationale

The paper's core algorithmic contribution — a state-dependent softmax mixture of horizon-specific critics trained by semi-gradient TD — is not itself circular: the gating weights are fitted to minimize undiscounted TD error, and the continual-learning results are measured against baselines. However, the paper's theoretical justification for the stronger claim that the mixture 'identifies effective discount factors' is Theorem 1. In that theorem, the key premise (Assumption 3) is logically equivalent to the conclusion (Eq. 36): the claim that the horizon maximizing undiscounted return is the one with smallest undiscounted Bellman residual is assumed, not derived. This is a load-bearing self-definitional step, because without it the theorem does not connect residual minimization to horizon selection. The paper's own Fig. 10 weakens the empirical support: for the Four Rooms task, five of ten seeds converge to weights favoring suboptimal discount factors, and the text concedes the mixture 'captures task-relevant structure beyond simply identifying an optimal discount factor.' Thus the central 'identification' claim is partially circular and empirically under-supported, while the broader mixture method retains independent empirical content. Score 6 reflects this partial circularity rather than a fully forced derivation.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

No new physical or theoretical entities are introduced; the gating network is a standard component. The central burden is the set of assumptions in Theorem 1, especially Assumption 3, which effectively postulates that Bellman-residual minimization identifies the return-optimal horizon. Hyperparameters are numerous and empirically fitted per task.

free parameters (6)
  • discount factor grid {1, 0.998, 0.996, 0.992, 0.984, 0.969, 0.938, 0.875, 0.75, 0.5} = 10 discrete values spanning effective horizons 2..∞
    The set of experts is fixed; the effective-horizon doubling is a design choice that could favor certain tasks. The grid itself is not learned.
  • trace parameter λ = 0.8
    Fixed for all experts and all tasks; no sensitivity analysis is reported.
  • per-task learning rates α = 0.001 (foraging), 0.1 (goal), 0.01 (four rooms)
    Explicitly 'empirically selected for each environment' (Section 4.1). These values strongly affect convergence; tuning them per task undercuts the claim of removing manual tuning.
  • task-switch interval = 4000 episodes
    'Empirically selected as a sufficient duration for each task to converge' (Section 4.1). Requires knowing when the task changes.
  • replay buffer update interval and buffer size = 50 steps; buffer size unstated
    The gating update consumes replay-buffer trajectories every 50 steps; buffer size and sampling scheme are not specified.
  • exploration schedule = ε: 1→0.05, decay 0.999; reset at each task
    Reset at task boundaries requires an external supervisory signal in the claimed continual setup.
axioms (5)
  • domain assumption Each critic Q_i converges to the true discounted Q-value for its γ_i under Expected SARSA(λ) with ε-greedy
    Invoked in Theorem 1 Assumption 1; not proven. ε decays to 0.05 (not zero), so asymptotic convergence to the optimal policy for fixed γ is not guaranteed; the critics are also coupled through the shared policy.
  • ad hoc to paper The discount factor maximizing undiscounted episodic return coincides with the discount factor minimizing the undiscounted Bellman residual (Assumption 3 of Theorem 1)
    This is the load-bearing premise of the Theorem; it asserts the conclusion of the theorem (that residual-minimizing weights select the return-optimal horizon). The paper's four-rooms results suggest it can fail.
  • ad hoc to paper Bellman residuals of different experts do not cancel in the convex combination (Assumption 4)
    Needed to prevent the mixture from having a smaller residual than any single expert without actually selecting the best horizon; unverified.
  • domain assumption A convex combination of discounted Q-functions can represent the undiscounted optimal value well enough for greedy action selection
    The mixture is scored with γ=1 TD error, yet each expert approximates a discounted return; the expressiveness of this restricted class is not analyzed.
  • domain assumption State representation φ(s)=(x, y, orientation) suffices for the gating network to identify the correct horizon
    Full-grid observations are provided to the agent; the gating input is claimed to be this low-dimensional sufficient statistic, but no ablation is given.

pith-pipeline@v1.3.0-alltime-deepseek · 8840 in / 14550 out tokens · 113452 ms · 2026-08-01T09:42:41.432504+00:00 · methodology

0 comments
read the original abstract

Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences. In reinforcement learning (RL), this trade-off is typically controlled through a fixed discount factor, which imposes a single exponentially discounted temporal horizon. However, biological agents exhibit flexible and adaptive temporal discounting, suggesting that effective planning requires multiple timescales. Here, we propose a multi-horizon approach that adaptively selects and combines temporal horizons, enabling robust adaptation to changes in reward structure without manual discount-factor tuning. This flexibility makes the method particularly suitable for continual learning scenarios involving task switches and varying environmental configurations. Empirically, we demonstrate that our approach identifies effective discount factors across a range of MiniGrid environments, including continual settings composed of three sequentially changing tasks. These results suggest that adaptive temporal discounting can improve parameter efficiency and enhance adaptability in both artificial and biologically inspired learning systems.

Figures

Figures reproduced from arXiv: 2607.20656 by Doina Precup, Manoosh Samiei, Paul Masset.

Figure 1
Figure 1. Figure 1: Fork Environments. Trap-jackpot fork is illustrated in Figure 1b, with corresponding Q-values reported in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Reward configurations for different reward dispersions [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance, measured by return and reward per step, as a function of the discount factor [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Return and reward per step, as a function of the discount factor ( [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Return and reward per step, as a function of the discount factor ( [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance metrics for different effective horizons in the foraging environment. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance metrics for different effective horizons in the goal reaching environment. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Performance metrics for different effective horizons in the four rooms environment. [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Learning performance during continual task transitions. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Mixer weights assigned to each discount factor during continual task switching. [PITH_FULL_IMAGE:figures/full_fig_p007_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Return and reward per step, as a function of the discount factor ( [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: The three tasks in the continual setup, presented to the agent in sequential order. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Grid layout with increasing Gaussian sampling variance (rows) and number of rewards [PITH_FULL_IMAGE:figures/full_fig_p014_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 5 canonical work pages

  1. [1]

    2018 , publisher =

    Reinforcement Learning: An Introduction , author =. 2018 , publisher =

    2018

  2. [2]

    Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS) , pages=

    Value-Decomposition Networks For Cooperative Multi-Agent Learning , author=. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS) , pages=. 2018 , organization=

    2018

  3. [3]

    Neural Computation , volume=

    Hierarchical Mixtures of Experts and the EM Algorithm , author=. Neural Computation , volume=. 1994 , publisher=

    1994

  4. [4]

    Journal of Machine Learning Research , volume=

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity , author=. Journal of Machine Learning Research , volume=

  5. [5]

    2022 , eprint=

    Towards Continual Reinforcement Learning: A Review and Perspectives , author=. 2022 , eprint=

    2022

  6. [6]

    Thirty-ninth Conference on Neural Information Processing Systems , year=

    Horizon Reduction Makes RL Scalable , author=. Thirty-ninth Conference on Neural Information Processing Systems , year=

  7. [7]

    Cruz and Kenway Louie and Daniel C

    Margarida Sousa and Pawel Bujalski and Bruno F. Cruz and Kenway Louie and Daniel C. McNamee and Joseph J. Paton , title =. Nature , year =

  8. [8]

    Loss Dynamics of Temporal Difference Reinforcement Learning , url =

    Bordelon, Blake and Masset, Paul and Kuo, Henry and Pehlevan, Cengiz , booktitle =. Loss Dynamics of Temporal Difference Reinforcement Learning , url =

  9. [9]

    Nature Neuroscience , year =

    Doya, Kenji , title =. Nature Neuroscience , year =

  10. [10]

    Psychological Bulletin , volume=

    Specious reward: a behavioral theory of impulsiveness and impulse control , author=. Psychological Bulletin , volume=. 1975 , doi=

    1975

  11. [11]

    Riedmiller , title =

    Volodymyr Mnih and Koray Kavukcuoglu and David Silver and Alex Graves and Ioannis Antonoglou and Daan Wierstra and Martin A. Riedmiller , title =. CoRR , volume =. 2013 , url =. 1312.5602 , timestamp =

    Pith/arXiv arXiv 2013

  12. [12]

    arXiv preprint arXiv:2211.13746 , year=

    Melting Pot 2.0 , author=. arXiv preprint arXiv:2211.13746 , year=. 2211.13746 , archivePrefix=

    Pith/arXiv arXiv

  13. [13]

    and Modayil, Joseph and Delp, Michael and Degris, Thomas and Pilarski, Patrick M

    Sutton, Richard S. and Modayil, Joseph and Delp, Michael and Degris, Thomas and Pilarski, Patrick M. and White, Adam and Precup, Doina , title =. The 10th International Conference on Autonomous Agents and Multiagent Systems - Volume 2 , pages =. 2011 , isbn =

    2011

  14. [14]

    Multi-time Models for Temporally Abstract Planning , url =

    Precup, Doina and Sutton, Richard S , booktitle =. Multi-time Models for Temporally Abstract Planning , url =

  15. [15]

    Sutton , abstract =

    Richard S. Sutton , abstract =. TD Models: Modeling the World at a Mixture of Time Scales , editor =. Machine Learning Proceedings 1995 , publisher =. 1995 , isbn =. doi:https://doi.org/10.1016/B978-1-55860-377-6.50072-4 , url =

    work page doi:10.1016/b978-1-55860-377-6.50072-4 1995

  16. [16]

    Psychological Bulletin , volume =

    Leonard Green and Joel Myerson , title =. Psychological Bulletin , volume =. 2004 , publisher =

    2004

  17. [17]

    BioRxiv , year=

    Striatal dopamine pulses follow a temporal discounting spectrum , author=. BioRxiv , year=. doi:10.1101/2021.10.31.466705 , url=

    work page doi:10.1101/2021.10.31.466705 2021

  18. [18]

    PLoS Computational Biology , volume=

    A hierarchy of time-scales and the brain , author=. PLoS Computational Biology , volume=. 2008 , publisher=. doi:10.1371/journal.pcbi.1000209 , url=

    doi:10.1371/journal.pcbi.1000209 2008

  19. [19]

    Proceedings of the Royal Society B: Biological Sciences , volume=

    On hyperbolic discounting and uncertain hazard rates , author=. Proceedings of the Royal Society B: Biological Sciences , volume=. 1998 , publisher=

    1998

  20. [20]

    arXiv preprint arXiv:2208.07004 , year=

    AI for Global Climate Cooperation: Modeling Global Climate Negotiations, Agreements, and Long-Term Cooperation in RICE-N , author=. arXiv preprint arXiv:2208.07004 , year=. 2208.07004 , archivePrefix=

    Pith/arXiv arXiv

  21. [21]

    Journal of Neuroscience , volume=

    Dopamine, time, and impulsivity in humans , author=. Journal of Neuroscience , volume=. 2010 , doi=

    2010

  22. [22]

    Journal of Artificial Intelligence Research (JAIR) , volume=

    Towards Continual Reinforcement Learning: A Review and Perspectives , author=. Journal of Artificial Intelligence Research (JAIR) , volume=. 2022 , doi=

    2022

  23. [23]

    Nature , volume=

    Multi-timescale reinforcement learning in the brain , author=. Nature , volume=. 2025 , doi=

    2025

  24. [24]

    Journal of neurophysiology , volume=

    Predictive reward signal of dopamine neurons , author=. Journal of neurophysiology , volume=. 1998 , publisher=. doi:10.1152/jn.1998.80.1.1 , url=

    doi:10.1152/jn.1998.80.1.1 1998

  25. [25]

    Journal of Risk and Uncertainty , volume =

    Partha Dasgupta , title =. Journal of Risk and Uncertainty , volume =. 2008 , publisher =

    2008

  26. [26]

    Psychological Bulletin , volume =

    George Ainslie , title =. Psychological Bulletin , volume =. 1975 , publisher =

    1975

  27. [27]

    Fisher and Steven Perlow and Lisa Sherman , title =

    Leonard Green and Ewin B. Fisher and Steven Perlow and Lisa Sherman , title =. Behaviour Analysis Letters , year =

  28. [28]

    Mazur , title =

    James E. Mazur , title =. Journal of the Experimental Analysis of Behavior , volume =. 1985 , publisher =

    1985

  29. [29]

    Journal of Economic Literature , volume =

    Shane Frederick and George Loewenstein and Ted O'Donoghue , title =. Journal of Economic Literature , volume =. 2002 , publisher =

    2002

  30. [30]

    From reinforcement learning models to psychiatric and neurological disorders , volume =

    Maia, Tiago and Frank, Michael , year =. From reinforcement learning models to psychiatric and neurological disorders , volume =. Nature neuroscience , doi =

  31. [31]

    Nature Neuroscience , volume=

    Dopamine transients follow a striatal gradient of reward time horizons , author=. Nature Neuroscience , volume=. 2024 , publisher=

    2024

  32. [32]

    , title =

    Sutton, Richard S. , title =. Machine Learning , volume =. 1988 , doi =

    1988

  33. [33]

    Watkins, Christopher J. C. H. and Dayan, Peter , title =. Machine Learning , volume =. 1992 , doi =

    1992

  34. [34]

    and Larochelle, Hugo , title =

    Fedus, William and Gelada, Carles and Bengio, Yoshua and Bellemare, Marc G. and Larochelle, Hugo , title =. arXiv preprint arXiv:1902.06865 , year =

    Pith/arXiv arXiv 1902

  35. [35]

    and Malik, A

    Masset, Pierre and Tano, Paul and Kim, Hyun R. and Malik, A. N. and Pouget, Alexandre and Uchida, Naoshige , title =. Nature , volume =. 2025 , doi =

    2025

  36. [36]

    Devon and Lanctot, Marc , title =

    Rashid, Tabish and Samvelyan, Mikayel and de Witt, Christian Schroeder and Sukhbaatar, Sainbayar and Hjelm, R. Devon and Lanctot, Marc , title =. Proceedings of the 35th International Conference on Machine Learning , editor =. 2018 , publisher =

    2018

  37. [37]

    Read , title =

    Schultz, Wolfram and Dayan, Peter and Montague, P. Read , title =. Science , volume =. 1997 , doi =

    1997

  38. [38]

    arXiv preprint arXiv:2307.11046 , year =

    Abel, David and Barreto, Andre and Van Roy, Benjamin and Precup, Doina and van Hasselt, Hado and Singh, Satinder , title =. arXiv preprint arXiv:2307.11046 , year =

    Pith/arXiv arXiv

  39. [39]

    arXiv preprint arXiv:1907.02908 , year =

    Hessel, Matteo and van Hasselt, Hado and Modayil, Joseph and Silver, David , title =. arXiv preprint arXiv:1907.02908 , year =

    Pith/arXiv arXiv 1907

  40. [40]

    Advances in Neural Information Processing Systems , volume =

    Anand, Nishant and Precup, Doina , title =. Advances in Neural Information Processing Systems , volume =. 2023 , url =

    2023

  41. [41]

    and Sutton, Richard S

    Singh, Satinder P. and Sutton, Richard S. , title =. Machine Learning , volume =. 1996 , doi =

    1996

  42. [42]

    Rupam and Yu, Huizhen and White, Martha and Sutton, Richard S

    Mahmood, A. Rupam and Yu, Huizhen and White, Martha and Sutton, Richard S. , title =. arXiv preprint arXiv:1507.01569 , year =

    Pith/arXiv arXiv

  43. [43]

    Proceedings of the 36th International Conference on Machine Learning , editor =

    Romoff, Joshua and Henderson, Peter and Touati, Ahmed and Brunskill, Emma and Pineau, Joelle and Ollivier, Yann , title =. Proceedings of the 36th International Conference on Machine Learning , editor =. 2019 , publisher =

    2019

  44. [44]

    and Barto, Andrew G

    Sutton, Richard S. and Barto, Andrew G. , publisher=. Reinforcement Learning:. 1998 , address=

    1998

  45. [45]

    R. S. Sutton and D. McAllester and S. Singh and Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. Advances in Neural Information Processing Systems 12. 2000

    2000

  46. [46]

    R. J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning. 1992

    1992

  47. [47]

    The Quarterly Journal of Economics , author =

    Golden. The Quarterly Journal of Economics , author =. 1997 , pages =. doi:10.1162/003355397555253 , number =

    doi:10.1162/003355397555253 1997

  48. [48]

    Journal of Experimental Psychology: Animal Learning and Cognition , author =

    Delay discounting:. Journal of Experimental Psychology: Animal Learning and Cognition , author =. 2016 , keywords =. doi:10.1037/xan0000097 , number =

    work page doi:10.1037/xan0000097 2016

  49. [49]

    PLoS ONE , author =

    Temporal-difference reinforcement learning with distributed representations , volume =. PLoS ONE , author =. 2009 , pages =. doi:10.1371/journal.pone.0007362 , abstract =

    work page doi:10.1371/journal.pone.0007362 2009

  50. [50]

    Science , author =

    Addiction as a computational process gone awry , volume =. Science , author =. 2004 , pages =. doi:10.1126/science.1102384 , abstract =

    work page doi:10.1126/science.1102384 2004