Avoiding Death through Fear Intrinsic Conditioning

Alexander Ororbia; Ferat Sahin; Jamison Heard; Rodney Sanchez

REVIEW 5 major objections 7 minor 38 references

Avoiding Death through Fear Intrinsic Conditioning

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A reinforcement learning agent can learn to avoid terminal 'death' states without direct experience, using an intrinsic fear reward produced by a memory-augmented network trained on a handful of dangerous behavior examples.

desk verdict Clever architecture, weak evaluation: the behavior-sequence fear reward is a plausible idea, but the danger classifier is never validated on the agent's own rollouts, so the central claim is unsupported. read the letter →

arxiv 2506.05529 v1 pith:WZYSHQKR submitted 2025-06-05 cs.AI cs.LG

Rodney Sanchez , Ferat Sahin , Alexander Ororbia , Jamison Heard This is my paper

classification cs.AIcs.LG

keywords intrinsicrewardfearconditioningmemory-augmentedneuralnetworkSiameseLSTMpartiallyobservableMarkovdecisionprocesssparseterminalstateavoidancegeneralizedanxietydisorder

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a reinforcement learning agent can learn to avoid terminal, 'death' states without ever experiencing them, by receiving an intrinsic fear signal inspired by social fear conditioning. The authors build a memory-augmented network, the Siamese MANN, trained on a small set of hand-crafted dangerous and safe behavior trajectories, and use its similarity scores as a negative reward that punishes behavior resembling the dangerous examples. In the MiniWorld Sidewalk POMDP, base PPO never reaches the goal, while fear-conditioned agents survive longer on average and sometimes find the goal. Varying the threshold at which fear is triggered reproduces a spectrum from over-cautious, anxiety-like avoidance to riskier exploration, which the authors connect to general anxiety disorder overgeneralization.

What carries the argument

The central object is the Siamese LSTM (SLSTM) controller embedded in a memory-augmented neural network (SMANN). The SLSTM replaces the linear projection gates of an LSTM with a Siamese (twin) network so the controller can mix image sequences and prior read vectors into one encoding; the SMANN then reads from memory by cosine similarity between the encoding of a behavior (state transitions) and stored parent behaviors. This read operation changes the MANN from comparing single states to comparing behaviors, and the resulting probability is scaled by a threshold to form the intrinsic reward.

What would settle it

Replace the trained SMANN's fear score with a fixed random score produced from the same 38 trajectories, retrain the agent, and check whether the episode-length and goal-finding differences disappear; if they remain, the observed avoidance is not caused by learned fear representations.

Watch

Extended reading notes

Core claim

The central claim is that an intrinsic reward derived from remembered behaviors—rather than from direct experience of danger—can deter an agent from non-descriptive terminal conditions and produce avoidance behavior resembling animal fear conditioning. The paper operationalizes social conditioning as a behavior value computed over state transitions supplied by a 'parent' (trust set to 1), stores these behavior encodings in a memory, and at each step penalizes the agent proportionally to the cosine similarity between its current behavior sequence and the stored dangerous behaviors. This makes the agent avoid a class of state transitions, not just a single stimulus state, and the threshold parameter controls how stringently the representation generalizes. Empirically, the fear-conditioned PPO agent achieves longer average episode lengths (up to 135.4 steps) than base PPO (115.1), and several runs collect positive extrinsic reward (maximum 0.082) where base PPO collects none.

Load-bearing premise

The whole scheme depends on the fear classifier, trained on just 38 examples of 'danger' and 'safe' behavior, correctly recognizing danger in the agent's own experiences; if the agent's view of the world and the classifier's view drift apart, the fear signal becomes noise.

Editorial extensions

If this is right

Agents can be preconditioned to avoid terminal states using only a handful of demonstrated dangerous behaviors, reducing the need to sample lethal states during training.
The threshold parameter gives a tunable safety-exploration tradeoff: lower thresholds produce longer, safer episodes but fewer goal findings, while higher thresholds allow more exploration and more successes.
The same framework could be applied to other partially-observable, sparse-reward environments with non-descriptive terminal conditions, such as the Lavagap environment.
The method suggests a route toward single-life RL, where an agent cannot afford to die during training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mechanism is genuine, the same learned fear signal could shape behavior in continuous control and robotics by conditioning on demonstrations rather than engineered reward penalties; this is our inference, not tested in the paper.
The threshold-behavior curve suggests a concrete testable analogue of anxiety: an agent with a near-zero threshold should freeze or become pathologically avoidant, and one with too high a threshold should fail to generalize danger; examining the shape of this curve across environments could ground the GAD analogy in measured performance.
A harder test the paper does not run: ablating the SLSTM's memory read and replacing the fear score with a random-but-fixed classifier trained on the same 38 trajectories would show whether avoidance comes from the learned behavior representation or merely from any constant penalty.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new piece is the SLSTM module that replaces linear projections in LSTM gates with a twin network, letting the controller mix image and vector modalities at the unit level. That is a real architectural contribution, and using behavior sequences rather than single states for the intrinsic fear signal extends their earlier stimuli-based work in a sensible way. The biological framing (amygdala, vicarious conditioning) is mostly motivational, but the underlying idea—low-shot dangerous demonstrations shape a reward penalty—is worth taking seriously.

The paper also deserves credit for being honest: they explicitly flag the 38-trajectory training set, the possibility of feature misalignment, and the limitations of PPO's exploration. The code is promised. That transparency helps.

The load-bearing soft spot is that the danger classifier is never validated on the policy's own state distribution. It is trained on 38 hand-crafted trajectories per class, generated by an oracle that forces the approach to the terminal condition. The authors acknowledge that the SMANN and PPO feature extractors may be misaligned, and since the SMANN is frozen during training, its danger scores on the agent's rollouts could be miscalibrated or arbitrary. If the intrinsic reward does not actually correlate with death proximity, the observed episode-length increases could be produced by any dense negative penalty that suppresses exploration. That is a reward-shaping story, not fear conditioning, and the missing baseline (reward shaping with a comparable dense penalty) makes it hard to know what the SLSTM is buying.

The empirical results are thin: 5 runs, no significance tests, and the abstract's claim that the agent 'solves' the environment is not supported by the numbers. Max extrinsic reward reaches 0.082, while the offline upper bound is 0.82. The threshold analysis is post hoc and reads as curve fitting to the GAD narrative.

In proportion: the architecture is new and the direction is promising, but the central mechanism is unverified and the evaluation is too weak to carry the conclusions. A serious referee could help the authors focus on the right experiments—classifier calibration on policy states, an ablation of the SLSTM, and a proper reward-shaping baseline.

Recommendation: deserved peer review, not desk rejection. The idea is interesting enough that a competent referee could turn it into something solid, but I would not cite it yet and I would only bring it to a reading group as a case study in how not to evaluate a safety signal.

Referee Report

5 major / 7 minor

Summary. The paper proposes an intrinsic reward mechanism for RL agents to avoid terminal states without directly experiencing them, inspired by social fear conditioning. The agent uses a Siamese MANN with a new SLSTM module trained on a small set of parent behavior trajectories (danger and non-danger classes) to compute a similarity score between the agent's current behavior and stored fear behaviors. A thresholded negative intrinsic reward is added to the extrinsic reward. Experiments in Miniworld Sidewalk, a POMDP with sparse rewards and a non-descriptive terminal condition, show that PPO with this fear intrinsic reward achieves longer average episode lengths than base PPO (e.g., 128.7 and 135.4 for low and mid thresholds vs. 115.1 for base PPO) and that some runs occasionally reach the goal (max extrinsic reward up to 0.082). The authors also vary the threshold to model behaviors described under general anxiety disorders.

Significance. If the mechanism works as claimed, the paper would offer a potentially useful direction for safe exploration in RL, avoiding dangerous states without direct sampling. The proposal of a biologically-inspired architecture for fear conditioning is intriguing, and the paper includes pseudocode and a link to code, which is a strength. However, the current evidence is insufficient to establish the central claim. The intrinsic reward is essentially the output of a supervised classifier trained on 38 hand-crafted trajectories per class, and the paper acknowledges possible feature misalignment between the policy and the classifier. Without validating the classifier's predictive validity on the agent's own state distribution, the observed behavior could be explained by a generic dense penalty. The experimental results are also statistically weak, with only 5 runs per condition and large variances. Thus, while the idea is potentially interesting, the present version does not convincingly demonstrate fear-conditioned avoidance behavior.

major comments (5)

[Section 4 and Limitations] The central claim that the fear intrinsic reward deters exploration of terminal states requires the SMANN's danger probability to be genuinely predictive of death on the agent's own rollout distribution. The paper provides no such validation. The SMANN is trained on only 38 hand-crafted trajectories per class (Section 4) and is frozen during PPO training, while the Limitations section acknowledges "possible misalignment of the agents' feature extractor and the SMANN feature extractor." If the classifier's scores are not predictive of actual terminal states on the policy's state distribution, the observed increase in episode length (Table 1) could result from any dense negative reward reducing exploration, not from behavior-specific fear conditioning. The authors should evaluate the SMANN's discrimination (e.g., ROC curves, calibration) on rollouts from the trained policy and, if necessary, re-train or adapt the classifier.
[Section 3, Eqs. (1)-(4)] The formal definition of behavior value is not rigorous and does not connect to the implemented intrinsic reward. Equations (1)-(4) mix set notation for transitions with value functions, use probabilities such as P(S''|*,S) that are not defined over a distribution of actions, and do not explain how V^π_{Parent} or V^π_{Peer} relate to the actual scalar intrinsic reward described in Algorithm 2 ("bad-behavior-prob * β-Value"). This makes it difficult to verify that the computational implementation matches the proposed theory. A precise specification of the intrinsic reward as a function of the SMANN output and the threshold is needed.
[Section 5, Table 1] The experimental results are statistically weak. Only 5 runs per condition are reported, and the differences in episode length overlap substantially across conditions (e.g., low β 128.701 ± 9.124 vs. base 115.061 ± 7.675). No significance tests are provided, and the text makes claims such as "the low threshold model was consistently able to find the goal" that are contradicted by the reported max extrinsic rewards (0.057 for low β, 0.026 for mid β). The claim that the agent can "solve" the environment relies on rare goal-reaching episodes (max extrinsic reward 0.082), which could be outliers. The authors should report more runs, include confidence intervals or statistical tests, and clearly define what constitutes solving the environment.
[Section 4] The training data for the SMANN is extremely small and hand-crafted: 38 trajectories per class, generated by an oracle that forces the agent to approach the terminal condition within three actions or to move away. The paper does not report the classifier's accuracy on any held-out set, nor does it analyze what features the SMANN learns. Without this, it is unclear whether the classifier is detecting risk of death or simply distinguishing scripted trajectories from random ones. The authors should provide classification performance metrics and an analysis of the intrinsic reward signal on the agent's actual rollouts.
[Section 5, Figure 4 and related work] The comparison with the stimuli-based baseline [29] is not sufficient to show that the proposed SLSTM/MANN mechanism provides a benefit beyond thresholding a classifier output. The stimuli method appears to give a constant negative reward (approximating a living cost), while the proposed method gates the reward by a threshold. To support the claim that the specific architecture and behavior-based similarity are important, the authors should compare against a simpler reward-shaping baseline that applies a negative penalty when a classifier trained on the same data deems the current state dangerous. Otherwise, the observed improvement may be attributable solely to the thresholding scheme rather than to the proposed memory architecture.

minor comments (7)

[General] The manuscript contains numerous grammatical and typographical errors, including the title "A voiding Death" (likely a formatting issue), "polices" instead of "policies" (Section 3), and incomplete sentence in Figure 7 caption ("increasing while still punishing the pristine representation of fear"). A thorough proofreading pass is needed.
[Section 3, Eqs. (1) and (3)] Equations (1) and (3) are nearly identical except for the subscript, and the notation is inconsistent (e.g., P(a1|S) vs. P(a1|s); P(S'|a1, s) vs. P(S''|a1, s)). The authors should unify the notation and clearly define all variables, including the meaning of the set notation and the role of actions in behavior value.
[Appendix, Algorithms 2 and 4] The pseudocode references "β-Value" in Algorithm 2 without defining it; it is presumably the threshold β described in the main text, but the connection should be made explicit. Also, the algorithm numbering (2 and 4) does not align with the cross-references in the text; please ensure consistency.
[Section 5, Table 1] The row "Agent & Offline" is incomplete and unclear: the episode length and intrinsic reward columns are "NA", and the values 0.769 ± 0.060 and 0.820 are not labeled. Clarify that this is a separate offline method and is not directly comparable.
[Section 4] The description of threshold ranges is inconsistent: the text says low thresholds ranged from 0.25 to 0.45, mid from 0.5 to 0.75, and high from 0.75 to 0.95, but Table 1 reports only three specific thresholds (0.25, 0.60, 0.95). Clarify whether the ranges were averaged or whether only representative thresholds were used.
[Section 5, Analysis] The statement "the low threshold model was consistently able to find the goal whereas the middle threshold only found it once for a few episodes" is not supported by the data in Table 1, where low β has a max extrinsic reward of 0.057 and mid β has 0.026. Please reconcile this discrepancy or correct the claim.
[Abstract and conclusion] The abstract claims the agent is "able to solve environments with non-descriptive terminal conditions," but the experiments show only a few episodes with positive extrinsic reward. This overstates the results; please temper the claim or provide stronger evidence of consistent goal-reaching.

Circularity Check

1 steps flagged · score 6.0 of 10

The avoidance behavior is enforced by an intrinsic reward defined as the SMANN's fitted bad-behavior probability, so the central 'death avoidance' claim reduces to the reward construction.

fitted input called prediction [Section 4 (Experimental Setup) and Appendix Algorithm 2]
"The fear behavior dataset was produced by starting the agent at a non-fear state, then, within three actions, the agent was made to approach the terminal condition. ... Intrinsic reward= bad-behavior-prob*β-Value; Agent Reward=(Intrinsic reward+Extrinsic reward)"

The SMANN is trained to classify behavior sequences as fear/non-fear using a dataset in which the 'fear' class is generated by forcing the agent to approach the terminal condition. At deployment, the PPO agent's reward is exactly the SMANN's bad-behavior probability times beta (Algorithm 2). Minimizing this reward is therefore, by construction, a penalty for reproducing the hand-labeled terminal-approaching transitions. So the paper's claim that this intrinsic reward 'deters exploration of terminal states' is not an emergent prediction but the direct optimization objective encoded in the reward; the fitted danger labels are presented as observed avoidance.

full rationale

The core derivation chain is: hand-crafted fear trajectories -> SMANN classifier -> intrinsic reward -> avoidance. The last step is largely enforced by the reward definition, making the headline 'prediction' of death avoidance partially circular (score 6). The paper does not rely on a self-citation chain for its force; reference [29] is used as a baseline and data-creation precedent but is not load-bearing. The threshold-dependent behavioral differences (low vs. mid vs. high beta) are mostly consequences of the threshold's definition, though they are demonstrated empirically and are not the same as the central death-avoidance claim. The acknowledged feature-misalignment limitation is a validity risk rather than circularity, but it underscores that the only genuine empirical question—whether the frozen classifier transfers to the policy's own state distribution—is left open. No other self-defined equations or imported uniqueness theorems appear.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The central claim depends on several hand-chosen parameters and domain assumptions. The fear threshold beta and the trust factor kappa are set manually. The behavior classifier is a supervised model trained on a small expert-curated dataset, and the entire approach assumes that this classifier generalizes to the agent's experience. The SLSTM and SMANN are new computational components with no external evidence beyond this paper.

free parameters (3)

Fear threshold beta = 0.25, 0.60, 0.95 (low, mid, high)
The threshold that determines when the intrinsic fear reward is applied. The paper varies it to emulate different anxiety levels, and the central results (episode length, goal finding) depend on this choice. It is not fitted to data but is a hand-chosen knob that controls the behavior.
Trust factor kappa = 1
The reliability of the peer/parent is set to 1 for all states, simplifying the model. This is an ad hoc assumption that the parental conditioning data is perfectly trusted.
SLSTM/MANN hyperparameters = learning rate 1e-3, epochs 150/300, memory size 128x40, etc.
These are architecture and training choices that could affect the classifier's accuracy and thus the intrinsic reward. They are not central to the claim but are free parameters in the pipeline.

assumptions (3)

domain assumption The hand-crafted 'fear behavior' dataset accurately represents behaviors that lead to the terminal condition.
The entire intrinsic reward hinges on the classifier learning from 38 trajectories per class that are labeled based on the experimenter's knowledge of the environment (approaching the sidewalk is dangerous). If these labels are wrong or unrepresentative, the reward is meaningless.
domain assumption Cosine similarity in the learned embedding space is a valid measure of behavioral similarity.
The MANN read operation compares keys using cosine similarity. The paper does not justify that this similarity correlates with actual danger across different state distributions.
domain assumption PPO with a fixed update rate can optimize the composite reward (extrinsic plus intrinsic) to produce meaningful avoidance.
The paper uses PPO and attributes failures to PPO's limited exploration. The interaction between the shaped reward and PPO is not analyzed; it is assumed that the intrinsic reward will steer the policy as intended.

invented entities (2)

Siamese LSTM (SLSTM) module
purpose: To mix image and vector modalities inside an LSTM controller, allowing the MANN to process sequences of states and prior reads.
A new neural architecture proposed by the paper. No independent evidence beyond the experimental results is provided.
Siamese MANN (SMANN)
purpose: To compare the agent's current behavior sequence to the stored fear behaviors and produce an intrinsic reward.
Combines the SLSTM with a memory-augmented network. It is the core of the intrinsic reward mechanism.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Avoiding Death through Fear Intrinsic Conditioning." pith.science (2026). https://pith.science/paper/WZYSHQKR

@misc{pith2026250605529,
  author       = {Pith},
  title        = {Pith review of: Avoiding Death through Fear Intrinsic Conditioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WZYSHQKR}},
  note         = {Machine review of arXiv:2506.05529}
}

read the original abstract

Biological and psychological concepts have inspired reinforcement learning algorithms to create new complex behaviors that expand agents' capacity. These behaviors can be seen in the rise of techniques like goal decomposition, curriculum, and intrinsic rewards, which have paved the way for these complex behaviors. One limitation in evaluating these methods is the requirement for engineered extrinsic for realistic environments. A central challenge in engineering the necessary reward function(s) comes from these environments containing states that carry high negative rewards, but provide no feedback to the agent. Death is one such stimuli that fails to provide direct feedback to the agent. In this work, we introduce an intrinsic reward function inspired by early amygdala development and produce this intrinsic reward through a novel memory-augmented neural network (MANN) architecture. We show how this intrinsic motivation serves to deter exploration of terminal states and results in avoidance behavior similar to fear conditioning observed in animals. Furthermore, we demonstrate how modifying a threshold where the fear response is active produces a range of behaviors that are described under the paradigm of general anxiety disorders (GADs). We demonstrate this behavior in the Miniworld Sidewalk environment, which provides a partially observable Markov decision process (POMDP) and a sparse reward with a non-descriptive terminal condition, i.e., death. In effect, this study results in a biologically-inspired neural architecture and framework for fear conditioning paradigms; we empirically demonstrate avoidance behavior in a constructed agent that is able to solve environments with non-descriptive terminal conditions.

Figures

Figures reproduced from arXiv: 2506.05529 by the authors.

**Figure 2.** Demonstrates the Siamese network that constitutes the gate in our SLSTM Module, the inclusion of this network for the gates allows for the mixing of images and vectors. For a larger version, see the appendix. propose the Siamese LSTM (SLSTM) module (see [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** An overview of the SLSTM controller and MANN (see appendix for a larger version). [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (13 more)

**Figure 4.** Figure 4: The stimuli fear’s constant negative reward approximates a living cost, making it optimal [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: At a low threshold value, the intrinsic reward punishes any policy that approximates the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: A mid(dle) threshold fear value promotes the learning of a policy that neither optimizes for [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: The high fear threshold model engages in the most exploration, increasing returns while [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: An overview of the Siamese LSTM Controller and MANN. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9: Demonstrates the Siamese Network that is then used as a gate for the SIAMESE LSTM [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 10.** Figure 10: Depicts the social learning frameworks where attention and retention use low-shot learning [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 11.** Figure 11: These Results demonstrate the achieved episodic reward of base PPO and the achieved [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

**Figure 12.** Figure 12: The stimuli fear’s constant negative reward approximates a living cost, making it optimal [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

**Figure 13.** Figure 13: At a low threshold value, the intrinsic reward punishes any policy that approximates the [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

**Figure 14.** Figure 14: Mid threshold fear promotes a policy that neither optimizes for greater episode length [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

**Figure 15.** Figure 15: The High fear threshold allows for the most exploration, increasing while still punishing [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

**Figure 16.** Figure 16: This is an example of the .25 threshold fifth run where the agent moves towards a zero [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 33 canonical work pages

[29]

Fear based intrinsic reward as a barrier function for continuous reinforcement learning

Rodney Sanchez, Ferat Sahin, and Jamison Heard. Fear based intrinsic reward as a barrier function for continuous reinforcement learning. In 2024 19th Annual System of Systems Engineering Conference (SoSE), pages 140–146. IEEE, 2024

work page 2024
[1]

Chris Askew and Andy P. Field. The vicarious learning pathway to fear 40 years on. Clinical Psychology Review, 28(7):1249–1265, 10 2008

work page 2008
[2]

Distop: Discovering a topological representation to learn diverse and rewarding skills

Arthur Aubret, Laetitia Matignon, and Salima Hassas. Distop: Discovering a topological representation to learn diverse and rewarding skills. IEEE Transactions on Cognitive and Developmental Systems, 15(4):1905–1915, 2023

work page 1905
[3]

Openai gym, 2016

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016

2016
[4]

Exploration by Random Network Distillation

Yuri Burda, Harrison Edwards, Amos J Storkey, and Oleg Klimov. Exploration by Random Network Distillation. ICLR, 2019

work page 2019
[5]

Ho Kei Cheng and Alexander G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022 , pages 640–658, Cham, 2022. Springer Nature Switzerland

work page 2022
[6]

Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks

Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo De Lazcano, Lucas Willems Mi- ple, Salem Lahlou, Pablo Samuel Castro, Google Deepmind, and Jordan Terry. Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks. Advances in Neural Information Processing Systems, 36, 2024

work page 2024
[7]

Craske, Dirk Hermans, and Bram Vervliet

Michelle G. Craske, Dirk Hermans, and Bram Vervliet. State-of-the-art and future directions for extinction as a translational model for fear and anxiety. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1742):20170025, 2018

work page 2018

Show all 38 references

[8]

Delgado, A

M.R. Delgado, A. Olsson, and E.A. Phelps. Extending animal models of fear conditioning to humans. Biological Psychology, 73(1):39–48, 2006. Translational Research: Parallels of Human and Animal Research in Biological Psychology

2006
[9]

Abnormal fear memory as a model for posttraumatic stress disorder

Aline Desmedt, Aline Marighetto, and Pier-Vincenzo Piazza. Abnormal fear memory as a model for posttraumatic stress disorder. Biological Psychiatry, 78(5):290–297, 2015. Understanding PTSD: From Mind to Molecules

2015
[10]

The infinite partially observable markov decision process

Finale Doshi-Velez. The infinite partially observable markov decision process. In Yoshua Bengio, Dale Schuurmans, John D. Lafferty, Christopher K. I. Williams, and Aron Culotta, editors, Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Inf...

2009
[11]

Gorman and Steven P

Jack M. Gorman and Steven P. Roose. The neurobiology of fear memory reconsolidation and psychoanalytic theory. Journal of the American Psychoanalytic Association, 59(6):1201–1220,
[12]

Neural turing machines

Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014

2014 arXiv
[13]

Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. Pmlr, 2018

2018
[14]

Craske, Susan Mineka, and Peter F

Dirk Hermans, Michelle G. Craske, Susan Mineka, and Peter F. Lovibond. Extinction in human fear conditioning. Biological Psychiatry, 60(4):361–368, 2006

2006
[15]

Neural networks and physical systems with emergent collective computational abilities

J J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci U S A, 79(8):2554–2558, April 1982

1982
[16]

Myers, Michael Davis, and Erica J

Tanja Jovanovic, Megan Keyes, Ana Fiallos, Karyn M. Myers, Michael Davis, and Erica J. Duncan. Fear potentiation and fear inhibition in a human fear-potentiated startle paradigm. Biological Psychiatry, 57(12):1559–1564, 2005

2005
[17]

Behavioral and neural mechanisms of overgeneral- ization in anxiety

Offir Laufer, David Israeli, and Rony Paz. Behavioral and neural mechanisms of overgeneral- ization in anxiety. Current Biology, 26(6):713–722, March 2016

2016
[18]

Learning subgoal representations with slow dynamics

Siyuan Li, Lulu Zheng, Jianhao Wang, and Chongjie Zhang. Learning subgoal representations with slow dynamics. In International Conference on Learning Representations, 2021

2021
[19]

Count-based exploration with the successor representation

Marlos C Machado, Marc G Bellemare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5125–5133, 2020

2020
[20]

Search-based adversarial estimates for improving sample efficiency in off-policy reinforcement learning

Federico Malato and Ville Hautamaki. Search-based adversarial estimates for improving sample efficiency in off-policy reinforcement learning. arXiv preprint arXiv:2502.01558, 2025

2025 arXiv
[21]

Online adaptation for enhancing imitation learning policies

Federico Malato and Ville Hautamäki. Online adaptation for enhancing imitation learning policies. In 2024 IEEE Conference on Games (CoG), pages 1–8, 2024

2024
[22]

Experimentally validated memris- tive memory augmented neural network with efficient hashing and similarity search

Ruibin Mao, Bo Wen, Arman Kazemi, Yahui Zhao, Ann Franchesca Laguna, Rui Lin, Ngai Wong, Michael Niemier, X Sharon Hu, Xia Sheng, et al. Experimentally validated memris- tive memory augmented neural network with efficient hashing and similarity search. Nature communications, 1...

2022
[23]

Herringa, and Mohammed R

Marie France Marin, Alexe Bilodeau-Houle, Simon Morand-Beaulieu, Alexandra Brouillard, Ryan J. Herringa, and Mohammed R. Milad. Vicarious conditioned fear acquisition and extinction in child–parent dyads. Scientific Reports, 10(1), 12 2020

2020
[24]

Milad, Christopher I

Mohammed R. Milad, Christopher I. Wright, Scott P. Orr, Roger K. Pitman, Gregory J. Quirk, and Scott L. Rauch. Recall of fear extinction in humans activates the ventromedial prefrontal cortex and hippocampus in concert. Biological Psychiatry, 62(5):446–454, 2007. Neurocircuitr...

2007
[25]

Phobias and preparedness: the selective, automatic, and encapsulated nature of fear

Susan Mineka and Arne Öhman. Phobias and preparedness: the selective, automatic, and encapsulated nature of fear. Biological Psychiatry, 52(10):927–937, 2002

2002
[26]

Rusu, Joel Veness, Marc G

V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstr...

2015
[27]

Towards robust shielded reinforcement learning through adaptive constraints and exploration: The fear field framework

Haritz Odriozola-Olalde, Maider Zamalloa, Nestor Arana-Arexolaleiba, and Jon Perez- Cerrolaza. Towards robust shielded reinforcement learning through adaptive constraints and exploration: The fear field framework. Engineering Applications of Artificial Intelligence , 144:110055, 2025

2025
[28]

Mortal computation: A foundation for biomimetic intelligence

Alexander Ororbia and Karl Friston. Mortal computation: A foundation for biomimetic intelligence. arXiv preprint arXiv:2311.09589, 2023. 11

2023 arXiv
[30]

Meta-Learning with Memory-Augmented Neural Networks, 6 2016

Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap, and Google Deepmind. Meta-Learning with Memory-Augmented Neural Networks, 6 2016

2016
[31]

Proximal policy optimization algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[32]

Planning to explore via self-supervised world models

Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International conference on machine learning, pages 8583–8592. PMLR, 2020

2020
[33]

Lewis, Andrew G

Satinder Singh, Richard L. Lewis, Andrew G. Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective.IEEE Transactions on Autonomous Mental Development, 2(2):70–82, 2010

2010
[34]

Neurobiology of infant fear and anxiety: Impacts of delayed amygdala development and attachment figure quality

Regina M Sullivan and Maya Opendak. Neurobiology of infant fear and anxiety: Impacts of delayed amygdala development and attachment figure quality. Biol Psychiatry, 89(7):641–650, August 2020

2020
[35]

Reinforcement Learning: An Introduction Second edition, in progress

Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction Second edition, in progress. MIT press, 2nd edition, 2018

2018
[36]

Sample efficient reinforce- ment learning via model-ensemble exploration and exploitation

Yao Yao, Li Xiao, Zhicheng An, Wanpeng Zhang, and Dijun Luo. Sample efficient reinforce- ment learning via model-ensemble exploration and exploitation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4202–4208. IEEE, 2021

2021
[37]

Computationally and sample efficient safe rein- forcement learning using adaptive conformal prediction

Hao Zhou, Yanze Zhang, and Wenhao Luo. Computationally and sample efficient safe rein- forcement learning using adaptive conformal prediction. international conference robotics and automation, 2025

2025
[38]

Importance weighted actor-critic for optimal conservative offline reinforcement learning.Advances in Neural Information Processing Systems, 36:49579–49602, 2023

Hanlin Zhu, Paria Rashidinejad, and Jiantao Jiao. Importance weighted actor-critic for optimal conservative offline reinforcement learning.Advances in Neural Information Processing Systems, 36:49579–49602, 2023. 12 7 Appendix / supplemental material 7.1 Larger Methodology Figu...

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [29]

Fear based intrinsic reward as a barrier function for continuous reinforcement learning

Rodney Sanchez, Ferat Sahin, and Jamison Heard. Fear based intrinsic reward as a barrier function for continuous reinforcement learning. In 2024 19th Annual System of Systems Engineering Conference (SoSE), pages 140–146. IEEE, 2024

work page 2024

[2] [1]

Chris Askew and Andy P. Field. The vicarious learning pathway to fear 40 years on. Clinical Psychology Review, 28(7):1249–1265, 10 2008

work page 2008

[3] [2]

Distop: Discovering a topological representation to learn diverse and rewarding skills

Arthur Aubret, Laetitia Matignon, and Salima Hassas. Distop: Discovering a topological representation to learn diverse and rewarding skills. IEEE Transactions on Cognitive and Developmental Systems, 15(4):1905–1915, 2023

work page 1905

[4] [3]

Openai gym, 2016

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016

2016

[5] [4]

Exploration by Random Network Distillation

Yuri Burda, Harrison Edwards, Amos J Storkey, and Oleg Klimov. Exploration by Random Network Distillation. ICLR, 2019

work page 2019

[6] [5]

Ho Kei Cheng and Alexander G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022 , pages 640–658, Cham, 2022. Springer Nature Switzerland

work page 2022

[7] [6]

Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks

Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo De Lazcano, Lucas Willems Mi- ple, Salem Lahlou, Pablo Samuel Castro, Google Deepmind, and Jordan Terry. Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks. Advances in Neural Information Processing Systems, 36, 2024

work page 2024

[8] [7]

Craske, Dirk Hermans, and Bram Vervliet

Michelle G. Craske, Dirk Hermans, and Bram Vervliet. State-of-the-art and future directions for extinction as a translational model for fear and anxiety. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1742):20170025, 2018

work page 2018