Pith. sign in

REVIEW 4 major objections 8 minor 53 references

Discounting and Drug Seeking in Biological Hierarchical Reinforcement Learning

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Temporal discounting can be built into hierarchical reinforcement-learning models of addiction, and doing so shows that faster discounting amplifies drug-seeking behavior at every level of the decision hierarchy.

desk verdict A plausible and useful extension of biological HRL to temporal discounting, but the central update rule is underived and the simulations are toy; worth review with revision. read the letter →

arxiv 2506.04549 v1 pith:QODQJC6T submitted 2025-06-05 q-bio.NC

Vardhan Palod , Pranav Mahajan , Veeky Baths , Boris S. Gutkin This is my paper
classification q-bio.NC
keywords hierarchicalreinforcementlearningtemporaldiscountingdrugseekingaddictiondopaminepredictionerrorstriatalhierarchyimpulsivitysubstanceusedisorder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that temporal discounting can be incorporated into a biologically structured hierarchical reinforcement-learning model of addiction, and that doing so explains why steeper discounting is associated with more severe drug seeking. The authors argue that a naive uniform discount factor distorts the value of natural rewards across hierarchy levels; their solution gives each level an adjusted discount factor and an 'undiscounting' step so natural reward values converge while drug reward values diverge. If the approach is right, addiction can be understood as a disorder of hierarchical decision-making in which lower, habitual levels gain control and over-discounting amplifies compulsive drug choices even when punishment follows.

What carries the argument

The load-bearing object is the adjusted discount factor combined with the undiscounting step in the temporal-difference update. Each level n gets a factor derived from the effective discount factor and the branching factor of the hierarchy, and the update divides the higher-level value by the higher-level discount factor raised to the number of steps to reward before multiplying by the current-level discount factor similarly. This makes the whole hierarchy follow one normative discount rate while keeping the inter-level coupling that models dopamine spirals, and it is the mechanism that produces convergence for natural rewards and divergence for drug rewards.

What would settle it

Record dopamine neuron activity in a task with hierarchical stimulus-reward structure and test whether the population signal tracks the number of primitive steps to a rewarded goal, separately at each abstraction level; if dopamine responses do not encode steps-to-reward as the model's nu(s,a) requires, the central mechanism fails. Behaviorally, the model predicts that in a two-choice task with equidistant food and drug rewards, lowering the discount factor from 0.9 to 0.6 increases the percentage of drug-seeking at every hierarchy level in the direction shown in Figure 4a, so re-running that task with a human or rodent agent and comparing choice shares would test the predicted monotonic amplification.

Watch

Extended reading notes

Core claim

The paper's central discovery is a discounting scheme for hierarchical reinforcement learning: the effective discount factor at the top level is redistributed into level-specific factors, and prediction errors are computed by undiscounting the higher-level value using steps-to-reward information before re-discounting at the current level. The authors claim this restores normative discounting for natural rewards, so Q-values for immediate and delayed food rewards converge across levels, while drug rewards, carrying a dopamine-induced bias in the prediction error, diverge across levels. Simulations show that increasing discounting (from gamma = 1 to 0.8 to 0.6) increases drug-seeking at every hierarchical level, with the effect largest when lower levels control behavior, matching the empirical link between steep delay discounting and addiction severity.

Load-bearing premise

The whole discounting construction depends on the agent knowing, at every moment at every level, exactly how many steps remain until the reward goal for the action it is taking; if the brain lacks that timing signal, the undiscounting step in Equation 5 cannot be executed and natural reward values would no longer converge across levels.

Editorial extensions

If this is right

  • If faster discounting amplifies drug seeking at all levels, then steep temporal discounting acts as an amplifier of hierarchical misvaluation, not merely a correlate; reducing discounting should shift choices away from drug seeking.
  • Because drug-seeking grows as behavioral control moves to lower levels, the model predicts that manipulations known to bias control downward, such as stress or drug re-exposure, should raise drug-seeking even if discounting is unchanged.
  • The selective-delay result implies that when natural rewards are delayed and drug rewards are immediate, low-level control produces drug-seeking regardless of the discount factor, so timing asymmetry combined with low-level control is sufficient for impulsive choice.
  • The model gives a mechanistic interpretation of the behavioral-marker literature: the discounting magnitude may index where in the hierarchy behavioral control sits, which could explain why abstinent opioid users remain impulsive while abstinent alcohol and nicotine users show recovery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that if chronic drug use itself steepens discounting, the model predicts a positive feedback loop: low-level drug misvaluation steepens discounting, which amplifies drug seeking, which further steepens discounting; this could be tested by letting the discount factor adapt with cumulative drug exposure.
  • Because the undiscounting step requires step-count information at every level, the model makes a testable neural prediction: midbrain dopamine signals should encode goal-distance or steps-to-reward information at each abstraction level, not only at cue presentation.
  • Mapping the hierarchy onto cortico-basal ganglia loops suggests that individual differences in discount steepness could be reinterpreted as differences in which level holds behavioral control; a soft-arbitration version would predict that the same discount factor yields different drug-seeking depending on arbitration costs, which could be compared with clinical trajectories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper extends a previously proposed hierarchical reinforcement learning (HRL) model of addiction (Keramati & Gutkin 2013; Mahajan et al. 2023) by introducing temporal discounting. The authors define level-dependent discount factors ζ(n) and an update rule (Eq. 5) that 'undiscounts' higher-level value information before re-discounting at the current level, and they simulate agents in simple two-choice MDPs. They report that for natural rewards, Q-values converge across hierarchy levels (Fig. 3), whereas for drug rewards, higher discounting (smaller γ) increases drug-seeking at every level (Figs. 4 and 5). The paper connects these results to the empirical association between steep delay discounting and addiction severity and discusses implications for hierarchical control and relapse.

Significance. If the core update rule is mathematically sound, the paper would fill a genuine gap: integrating discounting into biologically motivated HRL in a way that aligns with the known ventral-to-dorsal striatal dopaminergic spiral architecture. The qualitative match with the empirical discounting-addiction literature and the focus on hierarchical misvaluation are valuable. The paper is also honest about several limitations, including the biological plausibility of the 'undiscounting' step and the absence of a full account of addiction. However, the current evidence is predominantly a set of toy-environment simulations without statistical analysis, and—most importantly—the central algebraic mechanism (Eq. 5) is not derived or shown to have the claimed fixed points. The contribution is therefore promising but not yet established.

major comments (4)
  1. [Section 2.2, Eq. (5)] The formula as printed is ambiguous: it is unclear whether the factor (ζ(n+1))^{ν(s^{n+1}_t,a^{n+1}_t)} in the denominator divides the entire term (Q^{n+1}-r^{n+1}_t) or only r^{n+1}_t. Please rewrite Eq. 5 with explicit parentheses or displayed fractions. More substantively, this update rule is never derived from the semi-MDP/options formalism that the paper invokes in Section 2.1. To support the convergence claim, the authors should show that if Q^{n+1}(s,a) = ζ(n+1)^{ν(s,a)} R (the normative value at the higher level), then the update target for Q^n equals ζ(n)^{ν(s,a)} R. This requires a stated cross-level relation among the ν exponents (e.g., ν(s^n_t,a^n_t) = ϑ_n ν(s^{n+1}_t,a^{n+1}_t) plus a residual), which is absent. Without a fixed-point proof, the convergence in Fig. 3 and the drug-value divergence in Fig. 4 are not established by the text.
  2. [Section 4.4] The model's central mechanism requires the agent to know ν(s,a) at every level—the exact number of steps from the current state to the goal, measured in that level's temporal units. This is a strong computational assumption. The only cited evidence is a bioRxiv preprint (Sousa et al., 2023), and the Discussion itself concedes that the undiscounting step 'may reduce the biological plausibility of our model.' The authors should discuss how ν could be learned or approximated, or provide additional empirical grounding, because without this information the update rule cannot be implemented in a biologically plausible agent.
  3. [Sections 2.3 and 2.4, Figs. 4 and 5] The text claims that increasing discounting 'significantly increases drug-seeking at every level' and that 'these results are significant,' but no inferential statistics are reported. Figures 4 and 5 show only bar charts with error bars across 10 random seeds. Please provide appropriate statistical tests (e.g., a two-way ANOVA on level and discount factor, with multiple-comparison correction) or temper the language to 'descriptively increases.' The statistical claim is load-bearing for the abstract's central conclusion.
  4. [Sections 2.2 and 4.3] The convergence of natural-reward values is effectively built into the definition of ζ(n) in Eq. 3, and Section 4.3 states that 'the discount factors ζ(n) are selected to ensure that natural reward values remain consistent across hierarchical levels.' The demonstration in Fig. 3 is therefore a self-consistency check of the parameterization rather than an independent normative result. The paper would be strengthened by an explicit proposition stating the convergence property and a proof that Eq. 5 achieves it, distinct from the simulation.
minor comments (8)
  1. [Section 2.3] The phrase 'drug-seeking increases as we move down the hierarchy' should clarify the level numbering: since level 0 is the bottom and level L−1 the top, 'down' presumably means toward level 0, but this should be stated explicitly.
  2. [Section 2.1, Eq. (2)] Equation 2 has unbalanced parentheses; please check the intended grouping of γ with the bracketed term.
  3. [Figure 2b caption] The caption says 'The experiments were conducted in the two-choice MDP shown in (a)' but should state the number of episodes and random seeds, or refer readers to the Methods section.
  4. [Section 4.2] The term 'hard allocation' is used without defining the alternative; the 'soft allocation' scheme is mentioned only later in the same section. Please define both terms at first use.
  5. [Author list] The author list contains a stray space in 'V eeky Baths'; please correct to 'Veeky Baths'.
  6. [Sections 2.3 and 2.4] The word 'significantly' appears in several places without statistical support; consider replacing with 'notably' or 'descriptively' until formal tests are added.
  7. [Section 2.1] The phrase 'incorporating discounting into HRL discounting is challenging' contains a doubled word; please revise to 'incorporating discounting into HRL is challenging.'
  8. [Figures 4a and 5b] The figure captions should specify what the error bars represent (e.g., standard deviation across the 10 random seeds).

Circularity Check

1 steps flagged · score 6.0 of 10

Normative convergence of natural-reward Q-values is built into the ζ(n) construction; the drug-seeking amplification remains an emergent simulation.

  1. self definitional [Section 2.2 (Eqs. 3–5, Fig. 3); Methods Sections 4.3–4.4]
    "This allows us to propose an equation for the adjusted discount factors ζ(n) ... ensuring that values at different levels do not diverge, but rather converge according to the effective discount factor γ. ζ(n) =γ(QL−1 i=n 1 ϑi ) (3) ... We find that the HRL model based on equation 5 performs discounting normatively, as demonstrated by the results in Figure 3. The Q-values of actions leading to food rewards converge across all the levels in the hierarchy."

    The convergence reported in Fig. 3 is the design criterion used to define ζ(n), not an independent consequence. In the equal-ϑ case (Eq. 4), ζ(n)=γ^{(1/ϑ)^{L-1-n}}; with ν(s,a) defined as level-specific steps to goal, ζ(n)^{ν_n}=γ^{ν_{L-1}} (e.g., in Fig. 2a's 3-level ϑ=2 tree, ζ(0)^{ν_0}=ζ(1)^{ν_1}=γ when ν_0=2ν_1). Equation 5's undiscount (ζ(n+1)^{-ν}) and rediscount (ζ(n)^{ν}) factors are inserted specifically to cancel the level dependence of ζ, so the Q-values for natural rewards must come out equal across levels. Calling this a 'normative' finding is a restatement of the ansatz that ζ(n) was chosen to satisfy it; the only residual content is numerical convergence of the iterative updates, not a predicted equality.

full rationale

The paper's central addiction claim—higher discounting amplifies drug-seeking at all hierarchy levels (Figs. 4–5)—is not obtained by fitting the model to empirical data and is qualitatively consistent with the external delay-discounting literature (e.g., Bickel et al.), so that part has independent content. However, the 'normative discounting' result in Fig. 3, presented as a test of the model, is enforced by the choice of ζ(n) and the ν terms in Eq. 5; the paper explicitly says Eq. 3 exists to ensure values at different levels converge. This is an internal consistency check presented as a finding, i.e., a prediction that reduces by construction. The undiscounting step's biological plausibility is also self-flagged as a limitation, but that concern is about correctness, not additional circularity. No load-bearing uniqueness theorem or ansatz is imported from the authors' prior work; self-citations to Keramati and Gutkin and Mahajan et al. are background rather than the source of the discounting construction. Overall, partial circularity (6) because a major stated result reduces to its defining ansatz, while the clinically relevant drug-seeking prediction remains emergent.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The model introduces no new physical or theoretical entities; the 'stacked discounting map' is an algorithmic bookkeeping structure. The central claim rests on several domain assumptions from prior work and one strong ad hoc assumption, the steps-to-reward knowledge ν(s,a), which is load-bearing for the update rule.

free parameters (5)
  • effective discount factor γ = 1, 0.9, 0.8, 0.6
    Varied across simulations to study the effect of discounting on drug-seeking; not fitted to empirical data but is the key independent variable.
  • drug bias D = 3
    Chosen by scaling d = 2 from Mahajan et al. (2023) in proportion to the drug reward magnitude (+15 vs +10); affects the magnitude of drug-seeking.
  • branching factor ϑ = 2
    Number of lower-level options composing each higher-level option; determines the adjusted discount factors ζ(n).
  • learning rate α = 0.1
    Standard TD learning rate chosen following prior work; not fitted to the target result.
  • Boltzmann temperature = 10
    Exploration temperature used in action selection; follows Mahajan et al. (2023).
assumptions (5)
  • domain assumption The cortico-basal ganglia circuit implements hierarchical reinforcement learning with dopamine spirals coupling ventral to dorsal striatum.
    Invoked in Introduction and Eq. 1 following Haruno & Kawato (2006), Keramati & Gutkin (2013), Haber et al. (2000).
  • domain assumption Drug rewards add a non-negative bias d = +D to TD errors for drug-directed actions, while natural rewards have d = 0.
    Borrowed from Redish (2004), Dezfouli et al. (2009), Mahajan et al. (2023); used in Eq. 1 and throughout.
  • domain assumption Natural reward values should be consistent across all hierarchy levels; divergence is pathological and caused by drugs.
    Motivates the adjusted discount factor construction in Section 2.2.
  • ad hoc to paper The agent knows the number of steps to reward ν(s,a) at each level.
    Required for Eq. 5; acknowledged in Section 4.4 and Discussion as biologically uncertain.
  • domain assumption Rewards are delivered only upon completion of the abstract option and are then shared across levels.
    Follows Mahajan et al. (2023); stated in Methods Section 4.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Discounting and Drug Seeking in Biological Hierarchical Reinforcement Learning." pith.science (2026). https://pith.science/paper/QODQJC6T

@misc{pith2026250604549,
  author       = {Pith},
  title        = {Pith review of: Discounting and Drug Seeking in Biological Hierarchical Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QODQJC6T}},
  note         = {Machine review of arXiv:2506.04549}
}
read the original abstract

Despite a strong desire to quit, individuals with long-term substance use disorder (SUD) often struggle to resist drug use, even when aware of its harmful consequences. This disconnect between knowledge and compulsive behavior reflects a fundamental cognitive-behavioral conflict in addiction. Neurobiologically, differential cue-induced activity within striatal subregions, along with dopamine-mediated connectivity from the ventral to the dorsal striatum, contributes to compulsive drug-seeking. However, the functional mechanism linking these findings to behavioral conflict remains unclear. Another hallmark of addiction is temporal discounting: individuals with drug dependence exhibit steeper discount rates than non-users. Assuming the ventral-dorsal striatal organization reflects a gradient from cognitive to motor representations, addiction can be modeled within a hierarchical reinforcement learning (HRL) framework. However, integrating discounting into biologically grounded HRL remains an open challenge. In this work, we build on a model showing how action choices reinforced with drug rewards become insensitive to the negative consequences that follow. We address the integration of discounting by ensuring natural reward values converge across all levels in the HRL hierarchy, while drug rewards diverge due to their dopaminergic effects. Our results show that high discounting amplifies drug-seeking across the hierarchy, linking faster discounting with increased addiction severity and impulsivity. We demonstrate alignment with empirical findings on temporal discounting and propose testable predictions, establishing addiction as a disorder of hierarchical decision-making.

Figures

Figures reproduced from arXiv: 2506.04549 by the authors.

Figure 1
Figure 1. Comparison of Q-value updates in standard RL versus HRL with state abstraction. (a) In standard RL, the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. (a) The two-choice task environment where the agent chooses between a Food reward of [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The Q-values of actions leading to food rewards converge across all the levels in the hierarchy. The Q-value [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: This figure shows the simulation results of the two-choice task (Figure 2a) where drug reward is followed by [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: (a) The two-choice task environment where the agent chooses between a Food reward of [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Illustration of the Steps-to-Reward (ν) concept. In state S0, taking action c ′ (food action) leads directly to the food reward F in 1 step, so ν(S0, c′ ) = 1. However, taking action b (drug action) requires 3 steps to reach the drug reward D: S0 → S1 → S2 → D. Therefo…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 51 canonical work pages

  1. [1]

    Transition from moderate to excessive drug intake: change in hedonic set point

    Serge H Ahmed and GF Koob. Transition from moderate to excessive drug intake: change in hedonic set point. Science, 282 0 (5387): 0 298--300, 1998

    work page 1998

  2. [2]

    Specious reward: a behavioral theory of impulsiveness and impulse control

    George Ainslie. Specious reward: a behavioral theory of impulsiveness and impulse control. Psychological bulletin, 82 0 (4): 0 463, 1975

    work page 1975

  3. [3]

    Parallel organization of functionally segregated circuits linking basal ganglia and cortex

    Garrett E Alexander, Mahlon R DeLong, and Peter L Strick. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annual review of neuroscience, 9 0 (1): 0 357--381, 1986

    work page 1986

  4. [4]

    prefrontal

    Garrett E Alexander, Michael D Crutcher, and Mahlon R DeLong. Basal ganglia-thalamocortical circuits: parallel substrates for motor, oculomotor,“prefrontal” and “limbic” functions. Progress in brain research, 85: 0 119--146, 1991

    work page 1991

  5. [5]

    Steep delay discounting and addictive behavior: A meta-analysis of continuous associations

    Michael Amlung, Lana Vedelago, John Acker, Iris Balodis, and James MacKillop. Steep delay discounting and addictive behavior: A meta-analysis of continuous associations. Addiction, 112 0 (1): 0 51--62, 2017

    work page 2017

  6. [6]

    Delay discounting as a transdiagnostic process in psychiatric disorders: A meta-analysis

    Michael Amlung, Emma Marsden, Katherine Holshausen, Vanessa Morris, Herry Patel, Lana Vedelago, Katherine R Naish, Derek D Reed, and Randi E McCabe. Delay discounting as a transdiagnostic process in psychiatric disorders: A meta-analysis. JAMA psychiatry, 76 0 (11): 0 1176--1186, 2019

    work page 2019

  7. [7]

    Is the rostro-caudal axis of the frontal lobe hierarchical? Nature reviews neuroscience, 10 0 (9): 0 659--669, 2009

    David Badre and Mark D'esposito. Is the rostro-caudal axis of the frontal lobe hierarchical? Nature reviews neuroscience, 10 0 (9): 0 659--669, 2009

    work page 2009

  8. [8]

    Hierarchical cognitive control deficits following damage to the human frontal lobe

    David Badre, Joshua Hoffman, Jeffrey W Cooney, and Mark D'esposito. Hierarchical cognitive control deficits following damage to the human frontal lobe. Nature neuroscience, 12 0 (4): 0 515--522, 2009

    work page 2009

Show all 53 references
  1. [9]

    Cocaine seeking habits depend upon dopamine-dependent serial connectivity linking the ventral with the dorsal striatum

    David Belin and Barry J Everitt. Cocaine seeking habits depend upon dopamine-dependent serial connectivity linking the ventral with the dorsal striatum. Neuron, 57 0 (3): 0 432--441, 2008

    2008

  2. [10]

    Parallel and interactive learning processes within the basal ganglia: relevance for the understanding of addiction

    David Belin, Sietse Jonkman, Anthony Dickinson, Trevor W Robbins, and Barry J Everitt. Parallel and interactive learning processes within the basal ganglia: relevance for the understanding of addiction. Behavioural brain research, 199 0 (1): 0 89--102, 2009

    2009

  3. [11]

    Impulsivity and cigarette smoking: delay discounting in current, never, and ex-smokers

    Warren K Bickel, Amy L Odum, and Gregory J Madden. Impulsivity and cigarette smoking: delay discounting in current, never, and ex-smokers. Psychopharmacology, 146: 0 447--454, 1999

    1999

  4. [12]

    Behavioral and neuroeconomics of drug addiction: competing neural systems and temporal discounting processes

    Warren K Bickel, Michelle L Miller, Richard Yi, Benjamin P Kowal, Diana M Lindquist, and Jeffery A Pitcock. Behavioral and neuroeconomics of drug addiction: competing neural systems and temporal discounting processes. Drug and alcohol dependence, 90: 0 S85--S91, 2007

    2007

  5. [13]

    The behavioral-and neuro-economic process of temporal discounting: A candidate behavioral marker of addiction

    Warren K Bickel, Mikhail N Koffarnus, Lara Moody, and A George Wilson. The behavioral-and neuro-economic process of temporal discounting: A candidate behavioral marker of addiction. Neuropharmacology, 76: 0 518--527, 2014

    2014

  6. [14]

    Hierarchical models of behavior and prefrontal function

    Matthew M Botvinick. Hierarchical models of behavior and prefrontal function. Trends in cognitive sciences, 12 0 (5): 0 201--208, 2008

    2008

  7. [15]

    Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective

    Matthew M Botvinick, Yael Niv, and Andew G Barto. Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. cognition, 113 0 (3): 0 262--280, 2009

    2009

  8. [16]

    Dopamine, reinforcement learning, and addiction

    Peter Dayan. Dopamine, reinforcement learning, and addiction. Pharmacopsychiatry, 42 0 (S 01): 0 S56--S65, 2009

    2009

  9. [17]

    A neurocomputational model for cocaine addiction

    Amir Dezfouli, Payam Piray, Mohammad Mahdi Keramati, Hamed Ekhtiari, Caro Lucas, and Azarakhsh Mokri. A neurocomputational model for cocaine addiction. Neural computation, 21 0 (10): 0 2869--2893, 2009

    2009

  10. [18]

    Drugs abused by humans preferentially increase synaptic dopamine concentrations in the mesolimbic system of freely moving rats

    Gaetano Di Chiara and Assunta Imperato. Drugs abused by humans preferentially increase synaptic dopamine concentrations in the mesolimbic system of freely moving rats. Proceedings of the National Academy of Sciences, 85 0 (14): 0 5274--5278, 1988

    1988

  11. [19]

    Computational evidence for hierarchically structured reinforcement learning in humans

    Maria K Eckstein and Anne GE Collins. Computational evidence for hierarchically structured reinforcement learning in humans. Proceedings of the National Academy of Sciences, 117 0 (47): 0 29381--29389, 2020

    2020

  12. [20]

    Neural systems of reinforcement for drug addiction: from actions to habits to compulsion

    Barry J Everitt and Trevor W Robbins. Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nature neuroscience, 8 0 (11): 0 1481--1489, 2005

    2005

  13. [21]

    Animal models of addiction

    Eliot L Gardner. Animal models of addiction. In Textbook of Addiction Treatment: International Perspectives, pages 35--49. Springer, 2020

    2020

  14. [22]

    Model based control can give rise to devaluation insensitive choice

    Neil Garrett, Sean Allan, and Nathaniel D Daw. Model based control can give rise to devaluation insensitive choice. Addiction Neuroscience, 6: 0 100070, 2023

    2023

  15. [23]

    The reinforcing and subjective effects of morphine in post-addicts: a dose-response study

    SA Goldberg. The reinforcing and subjective effects of morphine in post-addicts: a dose-response study. Journal of Pharmacology and Experimental Therapeutics, 259: 0 1165--73, 1991

    1991

  16. [24]

    Liking and wanting of drug and non-drug rewards in active cocaine users: the strap-r questionnaire

    RZ Goldstein, PA Woicik, SJ Moeller, F Telang, M Jayne, C Wong, GJ Wang, JS Fowler, and ND Volkow. Liking and wanting of drug and non-drug rewards in active cocaine users: the strap-r questionnaire. Journal of psychopharmacology, 24 0 (2): 0 257--266, 2010

    2010

  17. [25]

    The primate basal ganglia: parallel and integrative networks

    Suzanne N Haber. The primate basal ganglia: parallel and integrative networks. Journal of chemical neuroanatomy, 26 0 (4): 0 317--330, 2003

    2003

  18. [26]

    Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum

    Suzanne N Haber, Julie L Fudge, and Nikolaus R McFarland. Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. Journal of Neuroscience, 20 0 (6): 0 2369--2382, 2000

    2000

  19. [27]

    Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fmri examination in stimulus-action-reward association learning

    Masahiko Haruno and Mitsuo Kawato. Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fmri examination in stimulus-action-reward association learning. Neural networks, 19 0 (8): 0 1242--1254, 2006

    2006

  20. [28]

    Per-decision option discounting

    Anna Harutyunyan, Peter Vrancx, Philippe Hamel, Ann Now \'e , and Doina Precup. Per-decision option discounting. In International Conference on Machine Learning, pages 2644--2652. PMLR, 2019

    2019

  21. [29]

    Safe state abstraction and discounting in hierarchical reinforcement learning

    Bernhard Hengst. Safe state abstraction and discounting in hierarchical reinforcement learning. University of New South Wales, School of Computer Science and Engineering, 2003

    2003

  22. [30]

    Addiction is driven by excessive goal-directed drug choice under negative affect: translational critique of habit and compulsion theory

    Lee Hogarth. Addiction is driven by excessive goal-directed drug choice under negative affect: translational critique of habit and compulsion theory. Neuropsychopharmacology, 45 0 (5): 0 720--735, 2020

    2020

  23. [31]

    The neural basis of addiction: a pathology of motivation and choice

    Peter W Kalivas and Nora D Volkow. The neural basis of addiction: a pathology of motivation and choice. American Journal of Psychiatry, 162 0 (8): 0 1403--1413, 2005

    2005

  24. [32]

    Imbalanced decision hierarchy in addicts emerging from drug-hijacked dopamine spiraling circuit

    Mehdi Keramati and Boris Gutkin. Imbalanced decision hierarchy in addicts emerging from drug-hijacked dopamine spiraling circuit. PloS one, 8 0 (4): 0 e61489, 2013

    2013

  25. [33]

    The architecture of cognitive control in the human prefrontal cortex

    Etienne Koechlin, Chrystele Ody, and Fr \'e d \'e rique Kouneiher. The architecture of cognitive control in the human prefrontal cortex. Science, 302 0 (5648): 0 1181--1185, 2003

    2003

  26. [34]

    Neurobiology of addiction

    George F Koob. Neurobiology of addiction. Addiction Counseling Review. New York2004, 2015

    2015

  27. [35]

    Cost-benefit arbitration between multiple reinforcement-learning systems

    Wouter Kool, Samuel J Gershman, and Fiery A Cushman. Cost-benefit arbitration between multiple reinforcement-learning systems. Psychological science, 28 0 (9): 0 1321--1333, 2017

    2017

  28. [36]

    Doing what’s not wanted: Conflict in incentives and misallocation of behavioural control can lead to drug-seeking despite adverse outcomes

    Pranav Mahajan, Veeky Baths, and Boris Gutkin. Doing what’s not wanted: Conflict in incentives and misallocation of behavioural control can lead to drug-seeking despite adverse outcomes. Addiction Neuroscience, 8: 0 100115, 2023. ISSN 2772-3925. doi:https://doi.org/10.1016/j.a...

    2023

  29. [37]

    The role of serotonin in the regulation of patience and impulsivity

    Katsuhiko Miyazaki, Kayoko W Miyazaki, and Kenji Doya. The role of serotonin in the regulation of patience and impulsivity. Molecular neurobiology, 45: 0 213--224, 2012

    2012

  30. [38]

    The role of serotonin in drug use and addiction

    Christian P M \"u ller and Judith R Homberg. The role of serotonin in drug use and addiction. Behavioural brain research, 277: 0 146--192, 2015

    2015

  31. [39]

    Delay discounting of money and alcohol in actively using alcoholics, currently abstinent alcoholics, and controls

    Nancy M Petry. Delay discounting of money and alcohol in actively using alcoholics, currently abstinent alcoholics, and controls. Psychopharmacology, 154: 0 243--250, 2001

    2001

  32. [40]

    Individual differences in nucleus accumbens dopamine receptors predict development of addiction-like behavior: a computational approach

    Payam Piray, Mohammad Mahdi Keramati, Amir Dezfouli, Caro Lucas, and Azarakhsh Mokri. Individual differences in nucleus accumbens dopamine receptors predict development of addiction-like behavior: a computational approach. Neural computation, 22 0 (9): 0 2334--2368, 2010

    2010

  33. [41]

    Temporal abstraction in reinforcement learning

    Doina Precup. Temporal abstraction in reinforcement learning. University of Massachusetts Amherst, 2000

    2000

  34. [42]

    Commitment, choice and self-control 1

    Howard Rachlin and Leonard Green. Commitment, choice and self-control 1. Journal of the experimental analysis of behavior, 17 0 (1): 0 15--22, 1972

    1972

  35. [43]

    Addiction as a computational process gone awry

    A David Redish. Addiction as a computational process gone awry. Science, 306 0 (5703): 0 1944--1947, 2004

    1944

  36. [44]

    Delay discounting, impulsiveness, and addiction severity in opioid-dependent patients

    Elias Robles, B Emma Huang, Pippa M Simpson, and Donald E McMillan. Delay discounting, impulsiveness, and addiction severity in opioid-dependent patients. Journal of substance abuse treatment, 41 0 (4): 0 354--362, 2011

    2011

  37. [45]

    Potential vulnerabilities of neuronal reward, risk, and decision mechanisms to addictive drugs

    Wolfram Schultz. Potential vulnerabilities of neuronal reward, risk, and decision mechanisms to addictive drugs. Neuron, 69 0 (4): 0 603--617, 2011

    2011

  38. [46]

    A neural substrate of prediction and reward

    Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science, 275 0 (5306): 0 1593--1599, 1997

    1997

  39. [47]

    Low-serotonin levels increase delayed reward discounting in humans

    Nicolas Schweighofer, Mathieu Bertin, Kazuhiro Shishida, Yasumasa Okamoto, Saori C Tanaka, Shigeto Yamawaki, and Kenji Doya. Low-serotonin levels increase delayed reward discounting in humans. Journal of Neuroscience, 28 0 (17): 0 4528--4532, 2008

    2008

  40. [48]

    Cruz, Kenway Louie, Daniel McNamee, and Joseph J

    Margarida Sousa, Pawel Bujalski, Bruno F. Cruz, Kenway Louie, Daniel McNamee, and Joseph J. Paton. Dopamine neurons encode a multidimensional probabilistic map of future reward. bioRxiv, 2023. doi:10.1101/2023.11.12.566727. URL https://www.biorxiv.org/content/early/2023/11/13/...

    2023 doi

  41. [49]

    Implicit cognition and addiction: a tool for explaining paradoxical behavior

    Alan W Stacy and Reinout W Wiers. Implicit cognition and addiction: a tool for explaining paradoxical behavior. Annual review of clinical psychology, 6 0 (1): 0 551--575, 2010

    2010

  42. [50]

    Dyna, an integrated architecture for learning, planning, and reacting

    Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2 0 (4): 0 160--163, 1991

    1991

  43. [51]

    Reinforcement learning: an introduction mit press

    Richard S Sutton and Andrew G Barto. Reinforcement learning: an introduction mit press. Cambridge, MA, 22447, 1998

    1998

  44. [52]

    Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model

    Yuji Takahashi, Geoffrey Schoenbaum, and Yael Niv. Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Frontiers in neuroscience, 2: 0 282, 2008

    2008

  45. [53]

    Hierarchies of planning and reinforcement learning for robot navigation

    Jan W \"o hlke, Felix Schmitt, and Herke van Hoof. Hierarchies of planning and reinforcement learning for robot navigation. In 2021 IEEE international conference on robotics and automation (ICRA), pages 10682--10688. IEEE, 2021

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.