Pith. sign in

REVIEW 4 major objections 4 minor 84 references

Deep reinforcement learning in World-Earth system models to discover sustainable management strategies

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A deep reinforcement learning agent can discover sustainable management policies for stylized World-Earth models, including a pathway that a prior viability-theory study deemed impossible.

desk verdict A genuine proof-of-concept for DRL in World-Earth models, with a headline about long-term sustainability that outruns the evidence; worth refereeing if the code and data appear. read the letter →

arxiv 1908.05567 v1 pith:RQ5YK2PA submitted 2019-08-15 physics.soc-ph cs.LG

Felix M. Strnad , Wolfram Barfuss , Jonathan F. Donges , Jobst Heitzig This is my paper · ORCID
classification physics.soc-phcs.LG
keywords deepreinforcementlearningWorld-EarthsystemmodelssustainablemanagementplanetaryboundariessocialfoundationsMarkovdecisionprocessesclimatechangemitigationtopologyof
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that deep reinforcement learning can be a practical method for discovering sustainable management strategies in model worlds where the human system and the Earth system co-evolve. The authors build an agent that observes the state of a World-Earth model, chooses among combinations of management options such as carbon taxes, renewable subsidies, nature protection and degrowth, and receives a reward only for staying within planetary boundaries and above social foundations. The agent learns, from this simple signal alone, policies that keep two stylized models inside the sustainable region—including a trajectory in the AYS model that a prior viability-theory algorithm had classified as impossible. The broader point is that a technique that needs no predefined welfare function and no state-space discretization might scale to more complex governance problems in the Earth system.

What carries the argument

The mechanism is the agent–environment interface recast as a Markov decision process, with a deep Q-network approximating the optimal action-value function $Q^*(s,a)$. The state is the model's variable vector, the actions are the available management combinations, and the reward is either a $1$ for staying inside the boundaries or a boundary-distance signal. Because the Q-function is approximated by a neural network, the agent can learn in continuous, high-dimensional state spaces without discretizing them, which is exactly what lets it move beyond grid-based viability approaches. The paper uses the topology-of-sustainable-management concepts of shelter and backwaters as the success criterion for trajectories.

What would settle it

Simulate one of the learned successful policies for a much longer horizon, say 5,000 years, from the same or nearby starting states; if any run crosses a planetary or social boundary after the original episode would have ended, the long-term sustainability claim collapses. A complementary check is to compute the forward-invariant shelter set of the model and see whether the endpoints of the agent's trajectories actually lie inside it.

Watch

Extended reading notes

Core claim

The central discovery is that a deep Q-learning agent, given only a survival or boundary-distance reward, can learn novel management policies that navigate two stylized World-Earth models into regions from which sustainability can be maintained. In the AYS model the agent finds a viable path from the current state to the shelter region that a prior discretized viability-theory study could not find; along the way it applies both degrowth and energy transformation, switching the energy transformation on and off near the boundaries to mimic a continuous tax and subsidy level. In the c:GLOBAL model the agent learns the decisive timing: renewable subsidies and nature protection run throughout, while the carbon tax is delayed until renewables have progressed enough and then switched off once their learning curve is passed. Under partial observability, even with only socio-economic variables, the agent still finds sustainable solutions, though more slowly and with a later decline in success that the authors trace to the composition of the replay buffer.

Load-bearing premise

The load-bearing premise is that reaching the TSM shelter within a 500-year simulation horizon—the paper's success criterion—guarantees indefinite sustainability, even though the authors report that seemingly converged trajectories sometimes violate boundaries at later times.

Editorial extensions

If this is right

  • In the AYS model, neither energy transformation nor degrowth alone suffices from the current state; both are needed, with degrowth used only for a limited period, to reach a shelter where management can be switched off.
  • In the c:GLOBAL model, the carbon tax must be timed: too early violates the social foundation, too late violates the planetary boundary; the learned policy switches it on only after renewables have advanced and off once the learning curve is passed.
  • The agent can learn with partial observations; even seeing only population, capital and renewable knowledge eventually yields successful policies, so perfect global monitoring may not be necessary.
  • Because the framework is formulated as a Markov decision process, it generalizes to stochastic, noisy, and multi-agent World-Earth models without changing the learning architecture.
  • Observational noise sharply decreases success, so practical applications of the method would need noise preprocessing or denoising of the state input.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The impossible-trajectory result is relative to the earlier algorithm's discrete grid and discrete action set; the agent's fast on–off switching approximates a continuous controller, so allowing continuous tax and subsidy levels would likely reveal a larger family of sustainable policies.
  • The agent's farsighted behavior—acting decades before the benefits appear—suggests the same machinery could be applied to higher-dimensional Earth system models, where grid-based viability methods cannot run at all.
  • The later decline in c:GLOBAL learning success points to replay-buffer composition as an underappreciated control knob; a direct test would be whether oversampling early timesteps prevents the forgetting the authors describe.
  • A rigorous long-term sustainability claim would require checking that the reached shelter states are forward-invariant under the model dynamics; if some are not, the finite-horizon result remains conditional.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a deep reinforcement learning (DRL) framework for discovering sustainable management strategies in stylized World-Earth system models. The authors formulate the agent-environment interaction as a Markov decision process, with discrete actions representing global governance measures such as carbon taxes, renewable-energy subsidies, nature protection, and degrowth. They apply the framework to the three-dimensional AYS model and the seven-dimensional copan:GLOBAL model, using a deep Q-network extended with double Q-learning, dueling networks, and prioritized experience replay. The reported results include learning curves, example trajectories, and an analysis of partial observability and observation noise. The central claims are that the DRL agent learns farsighted policies that navigate the system into a sustainable 'shelter' region, that an AYS trajectory is found that a previous viability-theory study deemed impossible, and that the timing of carbon taxation and renewable subsidies is crucial for long-term sustainability.

Significance. If the central claims are established, this paper would be a valuable proof-of-concept, showing that model-free deep reinforcement learning can serve as a scalable complement to viability theory for exploring sustainable management strategies in nonlinear, higher-dimensional World-Earth models. The explicit formulation of the MDP, the detailed model equations in the appendix, and the systematic comparison of DQN variants are useful contributions. However, the headline claim of long-term sustainability is not yet adequately supported, because the success criterion is an informally defined 'shelter' region whose invariance is not verified, and because the quantitative reporting is incomplete. The paper is likely to be of interest to the community if these gaps can be closed.

major comments (4)
  1. [Section IIIB2, Fig. 6] The success criterion for the c:GLOBAL experiments is 'reaching the shelter region where management can be turned off', but the shelter region is never defined for this seven-dimensional model, nor is its topology-of-sustainable-management (TSM) shelter property verified. The paper itself observes in Section IIIB2 that 'seemingly converged trajectories sometimes transgressed boundaries at much later times', which directly undermines the inference that a finite-horizon trajectory reaching an informally identified region is sustainable in the long term. To support the central claim, the authors should provide a formal definition of the c:GLOBAL shelter set, show that every trajectory starting in it remains within the sustainability boundaries for all future times under the relevant action set (or at least under the default action), and report how closely the terminal states of successful tests approach this set.
  2. [Section IIIB1, Fig. 4 and Fig. 6] The quantitative evidence is incomplete. For the AYS model, Fig. 4 reports '200 independent simulations that find a trajectory inside the boundaries', but the denominator is not given, so no success probability can be inferred; for c:GLOBAL, Fig. 6 reports success fractions with confidence bands, but the definition of a successful test depends on the unverified shelter criterion of the previous comment. The absence of code and data further prevents independent reproduction of the learning curves and example trajectories. Please report success rates with denominators for both models, specify the evaluation protocol (episode length, start-state distribution, number of seeds), and make the code and data available.
  3. [Section IIIB1 and Conclusion] The claim that the DRL agent finds 'novel, previously undiscovered policies' and an AYS trajectory 'deemed impossible' in the viability-theory study of Kittel et al. is not yet supported by a precise comparison. It is unclear whether the prior study proved nonexistence of viable trajectories under the same action set and boundaries, or merely failed to find them because of state-space discretization. The authors should specify what exactly was deemed impossible, whether the discovered trajectory lies outside the viability kernel, and why the comparison is meaningful despite the different numerical methods. Without this, the novelty claim remains vague.
  4. [Section III C, Fig. 6] The partial-observability results are presented as a robustness property of the method, but the training and evaluation protocol for each observation set is not sufficiently specified. In particular, it is not clear whether each observation set was trained with the same hyperparameters, reward function, and episode distribution, and whether the same trained policies are then evaluated on the full hidden-state dynamics. Since the success metric inherits the shelter-criterion issue from the first major comment, the robustness claim is contingent on resolving that issue. Please clarify the protocol for each observation set and report the corresponding success rates.
minor comments (4)
  1. [Section IID a] The sustainability boundary for the AYS model is stated as 'a planetary boundary A>A_PB = 345 GtC', which contradicts the appendix where 'A may stay below some threshold A_PB'; please correct this to A < A_PB (and similarly check the direction of the c:GLOBAL boundary descriptions).
  2. [Section IIB and Table I] The target-network update frequency is described as 'iteration steps' in the text and as 'episodes' in Table I; please make these consistent, since the distinction affects the interpretation of the hyperparameter.
  3. [Figs. 5 and 7 captions] The symbols E_B, E_F, and R are used in the figure captions but are not defined in the main text; please define these variables in the model description in the appendix.
  4. [Section IIIA and Fig. 2] The text says the exploration rate decays 'exponentially from 1 to 0.01 at a decay rate of lambda = 0.001' for the AYS model, but Table I lists a final exploration of 0.001 for c:GLOBAL; please clarify which final exploration value applies to each environment in the figure and table.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the DRL policies are genuine outputs of reward-based search in fixed model dynamics; the shelter-criterion issue is a validation gap, not a circular reduction.

full rationale

The paper's derivations are self-contained in the relevant circularity sense. The learned management policies are outputs of a deep Q-learning optimization against a reward signal (survival or boundary-distance) and the fixed differential-equation dynamics of the AYS and c:GLOBAL environments; no parameter is fitted to reproduce a target trajectory, and no predicted quantity is identical by construction to an input. The observation that the boundary-distance reward leads the agent toward the green fixpoint, where boundary distance is maximized, is a consequence of the chosen objective rather than a separate prediction, but this is standard goal-directed RL, not circular reasoning: the agent must still discover the timing and action sequence in a high-dimensional nonlinear system. The comparison with viability theory (ref. 21) is co-authored by one of the present authors, but it is not load-bearing because Fig. 3 displays the found trajectory independently; the citation only supports the contextual claim that another method did not find it. The TSM 'shelter' concept is used as an interpretation and as the success criterion in Fig. 6, but the paper does not verify the shelter's invariance for c:GLOBAL and itself notes in Section IIIB2 that 'seemingly converged trajectories sometimes transgressed boundaries at much later times.' That is a validity limitation of the finite-horizon evaluation, not a circular reduction of the outcome to the input. Accordingly, the central claim is not forced by self-citation or by definition, and the circularity score is low.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities or new model equations; its contribution is the RL-control framework. The free parameters are learning-algorithm hyperparameters, hand-tuned rather than fitted to data. The load-bearing assumptions are the validity of the two stylized models and the TSM-based success criterion.

free parameters (5)
  • discount factor gamma = 0.96 (both environments)
    Hand-tuned in hyperparameter search (Table I); controls how much the agent values future rewards, directly affecting the learned farsightedness emphasized in the paper.
  • exploration decay rate lambda = 0.001
    Hand-tuned exponential decay for epsilon-greedy exploration; learning success depends on this schedule.
  • replay memory size = 1e5
    Hand-tuned; the paper notes it is environment-dependent and below default values in some DRL algorithms.
  • target network update frequency = 100 (AYS), 200 (c:GLOBAL)
    Hand-tuned; affects learning stability.
  • Adam learning rate = 0.00025
    Taken from recommended values in refs 34 and 54, but still a free parameter of the learning algorithm.
assumptions (5)
  • domain assumption The AYS and copan:GLOBAL models capture the essential qualitative dynamics of the World-Earth system for exploring sustainability.
    The paper takes the model equations and parameters from Kittel et al. (ref 21) and Nitzbon et al. (ref 56), with the renewable-knowledge extension from Donges et al. (ref 42); all results are conditional on these models.
  • domain assumption The chosen planetary boundary (A_PB = 345 GtC) and social foundation thresholds (Y_SF, W_SF) define the safe and just operating space.
    Adopted from Rockstrom et al. and Raworth via the model papers; the reward functions and success criteria are built on these thresholds.
  • domain assumption The TSM 'shelter' region is a valid proxy for long-term sustainability.
    The success criterion in Fig. 6 requires reaching the shelter region from ref 60; the paper assumes that reaching this region guarantees indefinite boundary compliance, which is not proven here.
  • standard math Standard DRL theory and the Rainbow algorithm behave as claimed.
    The paper relies on Bellman equations and published DRL extensions (double Q-learning, dueling, prioritized experience replay) without formal verification; these are established but implementation-dependent.
  • ad hoc to paper Starting episodes from states randomly drawn around the current state is a meaningful test distribution.
    The training and evaluation protocol in Section IIe uses this start distribution; the paper does not test generalization across the full state space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep reinforcement learning in World-Earth system models to discover sustainable management strategies." pith.science (2026). https://pith.science/paper/RQ5YK2PA

@misc{pith2026190805567,
  author       = {Pith},
  title        = {Pith review of: Deep reinforcement learning in World-Earth system models to discover sustainable management strategies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RQ5YK2PA}},
  note         = {Machine review of arXiv:1908.05567}
}
read the original abstract

Increasingly complex, non-linear World-Earth system models are used for describing the dynamics of the biophysical Earth system and the socio-economic and socio-cultural World of human societies and their interactions. Identifying pathways towards a sustainable future in these models for informing policy makers and the wider public, e.g. pathways leading to a robust mitigation of dangerous anthropogenic climate change, is a challenging and widely investigated task in the field of climate research and broader Earth system science. This problem is particularly difficult when constraints on avoiding transgressions of planetary boundaries and social foundations need to be taken into account. In this work, we propose to combine recently developed machine learning techniques, namely deep reinforcement learning (DRL), with classical analysis of trajectories in the World-Earth system. Based on the concept of the agent-environment interface, we develop an agent that is generally able to act and learn in variable manageable environment models of the Earth system. We demonstrate the potential of our framework by applying DRL algorithms to two stylized World-Earth system models. Conceptually, we explore thereby the feasibility of finding novel global governance policies leading into a safe and just operating space constrained by certain planetary and socio-economic boundaries. The artificially intelligent agent learns that the timing of a specific mix of taxing carbon emissions and subsidies on renewables is of crucial relevance for finding World-Earth system trajectories that are sustainable on the long term.

Figures

Figures reproduced from arXiv: 1908.05567 by the authors.

Figure 1
Figure 1. FIG. 1. Using the agent-environment interface [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. FIG. 3. Dynamics of a stylized World-Earth system according [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. FIG. 4. Analysis of predominant management strategies in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: FIG. 6. Percentage of tests the agent passes successfully given [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: FIG. 8. Percentage of successful tests for environments with [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 7
Figure 7. Figure 7: FIG. 7. Exemplary trajectories for unsuccessful management [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

84 extracted references · 60 canonical work pages

  1. [1]

    + command, where the argument is the citation key mentioned above. +

  2. [2]

    The AIP styles for REV 4 include Bib \ style files +aipnum.bst+ and +aipauth.bst+, appropriate for numbered and author-year bibliographies, respectively

    + commands may be crafted by hand or, preferably, generated by using Bib . The AIP styles for REV 4 include Bib \ style files +aipnum.bst+ and +aipauth.bst+, appropriate for numbered and author-year bibliographies, respectively. REV 4 will automatically choose the style appropriate for the document's selected class options: the default is numerical, and y...

  3. [3]

    merlin.mbs apsrev4-1.bst 2010-07-25 4.21a (PWD, AO, DPC) hacked

    FUNCTION id.bst "merlin.mbs apsrev4-1.bst 2010-07-25 4.21a (PWD, AO, DPC) hacked" ENTRY address archive archivePrefix author bookaddress booktitle chapter collaboration doi edition editor eid eprint howpublished institution isbn issn journal key language month note number organization pages primaryClass publisher school SLACcitation series title translati...

    2010

  4. [4]

    merlin.mbs apsrmp4-1.bst 2010-07-25 4.21a (PWD, AO, DPC) hacked

    FUNCTION id.bst "merlin.mbs apsrmp4-1.bst 2010-07-25 4.21a (PWD, AO, DPC) hacked" ENTRY address archive archivePrefix author bookaddress booktitle chapter collaboration doi edition editor eid eprint howpublished institution isbn issn journal key language month note number organization pages primaryClass publisher school SLACcitation series title translati...

    2010

  5. [5]

    author author H. J. \ Schellnhuber ,\ title title ‘earth system’analysis and the second copernican revolution , \ @noop journal journal Nature \ volume 402 ,\ pages C19 ( year 1999 ) NoStop

    work page 1999

  6. [6]

    author author J. F. \ Donges , author R. Winkelmann , author W. Lucht , author S. E. \ Cornell , author J. G. \ Dyke , author J. Rockstr \"o m , author J. Heitzig , \ and\ author H. J. \ Schellnhuber ,\ title title Closing the loop: Reconnecting human dynamics to earth system science , \ @noop journal journal The Anthropocene Review \ volume 4 ,\ pages 15...

    work page 2017

  7. [7]

    Rockstr \"o m , author W

    author author J. Rockstr \"o m , author W. Steffen , author K. Noone , author . Persson , author F. S. \ Chapin III , author E. F. \ Lambin , author T. M. \ Lenton , author M. Scheffer , author C. Folke , author H. J. \ Schellnhuber , et al. ,\ title title A safe operating space for humanity , \ @noop journal journal Nature \ volume 461 ,\ pages 472 ( yea...

    work page 2009

  8. [8]

    Rockstr \"o m , author W

    author author J. Rockstr \"o m , author W. L. \ Steffen , author K. Noone , author . Persson , author F. S. \ Chapin III , author E. Lambin , author T. M. \ Lenton , author M. Scheffer , author C. Folke , author H. J. \ Schellnhuber , et al. ,\ title title Planetary boundaries: exploring the safe operating space for humanity , \ @noop journal journal Ecol...

    work page 2009

Show all 84 references
  1. [9]

    author author U. G. \ Assembly ,\ @noop title Transforming our world: The 2030 agenda for sustainable development , \ type Tech. Rep. \ ( institution United Nations ,\ year 2015 ) NoStop

    2015

  2. [10]

    author author U. N. F. C. \ on Climate Change ,\ @noop title Conference of the parties - adoption of the paris agreement , \ type Tech. Rep. \ ( institution United Nations ,\ year 2015 ) NoStop

    2015

  3. [11]

    author author J. M. \ Anderies , author S. R. \ Carpenter , author W. Steffen , \ and\ author J. Rockstr \"o m ,\ title title The topology of non-linear global carbon dynamics: from tipping points to planetary boundaries , \ @noop journal journal Environmental Research Letters...

    2013

  4. [12]

    Steffen , author K

    author author W. Steffen , author K. Richardson , author J. Rockstr \"o m , author S. E. \ Cornell , author I. Fetzer , author E. M. \ Bennett , author R. Biggs , author S. R. \ Carpenter , author W. De Vries , author C. A. \ De Wit , et al. ,\ title title Planetary boundaries...

    2015

  5. [13]

    Raworth ,\ title title A safe and just space for humanity: Can we live within the doughnut , \ @noop journal journal Oxfam Policy Pract

    author author K. Raworth ,\ title title A safe and just space for humanity: Can we live within the doughnut , \ @noop journal journal Oxfam Policy Pract. Clim. Change Resil \ volume 8 ( year 2012 ) NoStop

    2012

  6. [14]

    Rogelj , author D

    author author J. Rogelj , author D. Shindell , author K. Jiang , author S. Fifita , author P. Forster , author V. Ginzburg , author C. Handa , author H. Kheshgi , author S. Kobayashi , author E. Kriegler , et al. ,\ title title Mitigation pathways compatible with 1.5 c in the ...

    2018

  7. [15]

    Steffen , author J

    author author W. Steffen , author J. Rockstr \"o m , author K. Richardson , author T. M. \ Lenton , author C. Folke , author D. Liverman , author C. P. \ Summerhayes , author A. D. \ Barnosky , author S. E. \ Cornell , author M. Crucifix , author J. F. \ Donges , author I. Fet...

    doi

  8. [16]

    u ller-Hansen , author M. Schl \

    author author F. M \"u ller-Hansen , author M. Schl \"u ter , author M. M \"a s , author J. F. \ Donges , author J. J. \ Kolb , author K. Thonicke , \ and\ author J. Heitzig ,\ title title Towards representing human behavior and decision making in earth system models--an overv...

    2017

  9. [17]

    author author D. L. \ Kelly \ and\ author C. D. \ Kolstad ,\ title title Integrated assessment models for climate change control , \ @noop journal journal International yearbook of environmental and resource economics \ volume 2000 ,\ pages 171--197 ( year 1999 ) NoStop

    2000

  10. [18]

    u ssensch \

    author author C. Pahl-Wostl , author C. Schlumpf , author M. B \"u ssensch \"u tt , author A. Sch \"o nborn , \ and\ author J. Burse ,\ title title Models at the interface between science and society: impacts and options , \ @noop journal journal Integrated assessment \ volume...

    2000

  11. [19]

    author author M. R. \ Bussieck \ and\ author A. Meeraus ,\ title title General algebraic modeling system (gams) , \ in\ @noop booktitle Modeling languages in mathematical optimization \ ( publisher Springer ,\ year 2004 )\ pp.\ pages 137--157 NoStop

    2004

  12. [20]

    author author R. S. \ Pindyck ,\ title title The use and misuse of models for climate policy , \ @noop journal journal Review of Environmental Economics and Policy \ volume 11 ,\ pages 100--114 ( year 2017 ) NoStop

    2017

  13. [21]

    author author M. I. \ Kamien \ and\ author N. L. \ Schwartz ,\ @noop title Dynamic optimization: the calculus of variations and optimal control in economics and management \ ( publisher Courier Corporation ,\ year 2012 ) NoStop

    2012

  14. [22]

    Liang ,\ title title Climate modification directed by control theory , \ @noop journal journal arXiv preprint arXiv:0805.0541 \ ( year 2008 ) NoStop

    author author W. Liang ,\ title title Climate modification directed by control theory , \ @noop journal journal arXiv preprint arXiv:0805.0541 \ ( year 2008 ) NoStop

    2008 arXiv

  15. [23]

    Botta , author P

    author author N. Botta , author P. Jansson , \ and\ author C. Ionescu ,\ title title The impact of uncertainty on optimal emission policies , \ @noop journal journal Earth System Dynamics \ volume 9 ,\ pages 525--542 ( year 2018 ) NoStop

    2018

  16. [24]

    Deffuant \ and\ author N

    author author G. Deffuant \ and\ author N. Gilbert ,\ @noop title Viability and resilience of complex systems: concepts, methods and case studies from ecology and society \ ( publisher Springer Science & Business Media ,\ year 2011 ) NoStop

    2011

  17. [25]

    Kittel , author R

    author author T. Kittel , author R. Koch , author J. Heitzig , author G. Deffuant , author J.-D. \ Mathias , \ and\ author J. Kurths ,\ title title Operationalization of topology of sustainable management to estimate qualitatively different regions in state space , \ @noop jou...

    2017 arXiv

  18. [26]

    author author R. S. \ Sutton , author A. G. \ Barto , et al. ,\ @noop title Introduction to reinforcement learning ,\ Vol.\ volume 135 \ ( publisher MIT press Cambridge ,\ year 1998 ) NoStop

    1998

  19. [27]

    author author F. B. \ von der Osten ,\ title Intelligent decision-making in coupled socio-ecological systems ,\ @noop Ph.D. thesis ,\ school University of Melbourne ( year 2017 ) NoStop

    2017

  20. [28]

    Mnih , author K

    author author V. Mnih , author K. Kavukcuoglu , author D. Silver , author A. A. \ Rusu , author J. Veness , author M. G. \ Bellemare , author A. Graves , author M. Riedmiller , author A. K. \ Fidjeland , author G. Ostrovski , et al. ,\ title title Human-level control through d...

    2015

  21. [29]

    Mnih , author K

    author author V. Mnih , author K. Kavukcuoglu , author D. Silver , author A. Graves , author I. Antonoglou , author D. Wierstra , \ and\ author M. Riedmiller ,\ title title Playing atari with deep reinforcement learning , \ @noop journal journal arXiv preprint arXiv:1312.5602 ...

    2013 arXiv

  22. [30]

    Tesauro ,\ title title Temporal difference learning and td-gammon , \ @noop journal journal Communications of the ACM \ volume 38 ,\ pages 58--68 ( year 1995 ) NoStop

    author author G. Tesauro ,\ title title Temporal difference learning and td-gammon , \ @noop journal journal Communications of the ACM \ volume 38 ,\ pages 58--68 ( year 1995 ) NoStop

    1995

  23. [31]

    Arulkumaran , author M

    author author K. Arulkumaran , author M. P. \ Deisenroth , author M. Brundage , \ and\ author A. A. \ Bharath ,\ title title A brief survey of deep reinforcement learning , \ @noop journal journal arXiv preprint arXiv:1708.05866 \ ( year 2017 ) NoStop

    2017 arXiv

  24. [32]

    Li ,\ title title Deep reinforcement learning , \ @noop journal journal arXiv preprint arXiv:1810.06339 \ ( year 2018 ) NoStop

    author author Y. Li ,\ title title Deep reinforcement learning , \ @noop journal journal arXiv preprint arXiv:1810.06339 \ ( year 2018 ) NoStop

    2018 arXiv

  25. [33]

    author author C. J. C. H. \ Watkins ,\ title Learning from delayed rewards ,\ @noop Ph.D. thesis ,\ school King's College, Cambridge ( year 1989 ) NoStop

    1989

  26. [34]

    LeCun , author Y

    author author Y. LeCun , author Y. Bengio , \ and\ author G. Hinton ,\ title title Deep learning , \ @noop journal journal nature \ volume 521 ,\ pages 436 ( year 2015 ) NoStop

    2015

  27. [35]

    \ Lin ,\ @noop title Reinforcement learning for robots using neural networks , \ type Tech

    author author L.-J. \ Lin ,\ @noop title Reinforcement learning for robots using neural networks , \ type Tech. Rep. \ ( institution DTIC Document ,\ year 1993 ) NoStop

    1993

  28. [36]

    Silver , author A

    author author D. Silver , author A. Huang , author C. J. \ Maddison , author A. Guez , author L. Sifre , author G. Van Den Driessche , author J. Schrittwieser , author I. Antonoglou , author V. Panneershelvam , author M. Lanctot , et al. ,\ title title Mastering the game of go...

    2016

  29. [37]

    Silver , author T

    author author D. Silver , author T. Hubert , author J. Schrittwieser , author I. Antonoglou , author M. Lai , author A. Guez , author M. Lanctot , author L. Sifre , author D. Kumaran , author T. Graepel , author T. Lillicrap , author K. Simonyan , \ and\ author D. Hassabis ,\ ...

    doi

  30. [38]

    Hessel , author J

    author author M. Hessel , author J. Modayil , author H. Van Hasselt , author T. Schaul , author G. Ostrovski , author W. Dabney , author D. Horgan , author B. Piot , author M. Azar , \ and\ author D. Silver ,\ title title Rainbow: Combining improvements in deep reinforcement l...

    2018

  31. [39]

    Mao , author M

    author author H. Mao , author M. Alizadeh , author I. Menache , \ and\ author S. Kandula ,\ title title Resource management with deep reinforcement learning , \ in\ @noop booktitle Proceedings of the 15th ACM Workshop on Hot Topics in Networks \ ( organization ACM ,\ year 2016...

    2016

  32. [40]

    Zhou , author X

    author author Z. Zhou , author X. Li , \ and\ author R. N. \ Zare ,\ title title Optimizing chemical reactions with deep reinforcement learning , \ @noop journal journal ACS central science \ volume 3 ,\ pages 1337--1344 ( year 2017 ) NoStop

    2017

  33. [41]

    author author T. P. \ Lillicrap , author J. Hunt , author A. Pritzel , author N. Heess , author T. Erez , author Y. Tassa , author D. Silver , \ and\ author D. Wierstra ,\ title title Continuous control with deep reinforcement learning (2015) , \ @noop journal journal arXiv pr...

    2015 arXiv

  34. [42]

    Levine , author C

    author author S. Levine , author C. Finn , author T. Darrell , \ and\ author P. Abbeel ,\ title title End-to-end training of deep visuomotor policies , \ @noop journal journal The Journal of Machine Learning Research \ volume 17 ,\ pages 1334--1373 ( year 2016 ) NoStop

    2016

  35. [43]

    Zhu , author R

    author author Y. Zhu , author R. Mottaghi , author E. Kolve , author J. J. \ Lim , author A. Gupta , author L. Fei-Fei , \ and\ author A. Farhadi ,\ title title Target-driven visual navigation in indoor scenes using deep reinforcement learning , \ in\ @noop booktitle 2017 IEEE...

    2017

  36. [44]

    Gu , author E

    author author S. Gu , author E. Holly , author T. Lillicrap , \ and\ author S. Levine ,\ title title Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates , \ in\ @noop booktitle 2017 IEEE international conference on robotics and automation ...

    2017

  37. [45]

    Haarnoja , author A

    author author T. Haarnoja , author A. Zhou , author S. Ha , author J. Tan , author G. Tucker , \ and\ author S. Levine ,\ title title Learning to walk via deep reinforcement learning , \ @noop journal journal arXiv preprint arXiv:1812.11103 \ ( year 2018 ) NoStop

    2018 arXiv

  38. [46]

    author author J. F. \ Donges , author J. Heitzig , author W. Barfuss , author J. A. \ Kassel , author T. Kittel , author J. J. \ Kolb , author T. Kolster , author F. M \"u ller-Hansen , author I. M. \ Otto , author M. Wiedermann , et al. ,\ title title Earth system modelling w...

    2018

  39. [47]

    author author W. B. \ Arthur ,\ title title Designing economic agents that act like human agents: A behavioral approach to bounded rationality , \ @noop journal journal The American Economic Review \ volume 81 ,\ pages 353--359 ( year 1991 ) NoStop

    1991

  40. [48]

    Lindkvist \ and\ author J

    author author E. Lindkvist \ and\ author J. Norberg ,\ title title Modeling experiential learning: The challenges posed by threshold dynamics for sustainable renewable resource management , \ @noop journal journal Ecological economics \ volume 104 ,\ pages 107--118 ( year 2014...

    2014

  41. [49]

    Lindkvist , author \"O

    author author E. Lindkvist , author \"O . Ekeberg , \ and\ author J. Norberg ,\ title title Strategies for sustainable management of renewable resources during environmental change , \ @noop journal journal Proc. R. Soc. B \ volume 284 ,\ pages 20162762 ( year 2017 ) NoStop

    2017

  42. [50]

    Rolnick , author P

    author author D. Rolnick , author P. L. \ Donti , author L. H. \ Kaack , author K. Kochanski , author A. Lacoste , author K. Sankaran , author A. S. \ Ross , author N. Milojevic-Dupont , author N. Jaques , author A. Waldman-Brown , et al. ,\ title title Tackling climate change...

    1906 arXiv

  43. [51]

    Barfuss , author J

    author author W. Barfuss , author J. F. \ Donges , \ and\ author J. Kurths ,\ title title Deterministic limit of temporal difference reinforcement learning for stochastic games , \ https://link.aps.org/doi/10.1103/PhysRevE.99.043305 journal journal Phys. Rev. E \ volume 99 ,\ ...

    doi

  44. [52]

    Wiering \ and\ author M

    author author M. Wiering \ and\ author M. Van Otterlo ,\ title title Reinforcement learning , \ @noop journal journal Adaptation, learning, and optimization \ volume 12 ,\ pages 51 ( year 2012 ) NoStop

    2012

  45. [53]

    Bellman ,\ title title A markovian decision process , \ @noop journal journal Journal of Mathematics and Mechanics \ ,\ pages 679--684 ( year 1957 ) NoStop

    author author R. Bellman ,\ title title A markovian decision process , \ @noop journal journal Journal of Mathematics and Mechanics \ ,\ pages 679--684 ( year 1957 ) NoStop

    1957

  46. [54]

    Van Hasselt , author A

    author author H. Van Hasselt , author A. Guez , \ and\ author D. Silver ,\ title title Deep reinforcement learning with double q-learning , \ booktitle booktitle Thirtieth AAAI Conference on Artificial Intelligence ,\ @noop journal journal Proceedings ofthe AAAI Conference on ...

    2016

  47. [55]

    author author H. V. \ Hasselt ,\ title title Double q-learning , \ in\ @noop booktitle Advances in Neural Information Processing Systems \ ( year 2010 )\ pp.\ pages 2613--2621 NoStop

    2010

  48. [56]

    Wang , author T

    author author Z. Wang , author T. Schaul , author M. Hessel , author H. Van Hasselt , author M. Lanctot , \ and\ author N. De Freitas ,\ title title Dueling network architectures for deep reinforcement learning , \ @noop journal journal arXiv preprint arXiv:1511.06581 \ ( year...

    2015 arXiv

  49. [57]

    author author M. G. \ Bellemare , author W. Dabney , \ and\ author R. Munos ,\ title title A distributional perspective on reinforcement learning , \ in\ @noop booktitle Proceedings of the 34th International Conference on Machine Learning-Volume 70 \ ( organization JMLR. org ,...

    2017

  50. [58]

    Schaul , author J

    author author T. Schaul , author J. Quan , author I. Antonoglou , \ and\ author D. Silver ,\ title title Prioritized experience replay , \ @noop journal journal arXiv preprint arXiv:1511.05952 \ ( year 2015 ) NoStop

    2015 arXiv

  51. [59]

    Barfuss , author J

    author author W. Barfuss , author J. F. \ Donges , author S. J. \ Lade , \ and\ author J. Kurths ,\ title title When optimization for governing human-environment tipping elements is neither sustainable nor safe , \ @noop journal journal Nature communications \ volume 9 ,\ page...

    2018

  52. [60]

    Nitzbon , author J

    author author J. Nitzbon , author J. Heitzig , \ and\ author U. Parlitz ,\ title title Sustainability, collapse and oscillations in a simple world-earth model , \ @noop journal journal Environmental Research Letters \ volume 12 ,\ pages 074020 ( year 2017 ) NoStop

    2017

  53. [61]

    Heitzig , author W

    author author J. Heitzig , author W. Barfuss , \ and\ author J. Donges ,\ title title A thought experiment on sustainable management of the earth system , \ @noop journal journal Sustainability \ volume 10 ,\ pages 1947 ( year 2018 ) NoStop

    1947

  54. [62]

    Zhang \ and\ author R

    author author S. Zhang \ and\ author R. S. \ Sutton ,\ title title A deeper look at experience replay , \ @noop journal journal arXiv preprint arXiv:1712.01275 \ ( year 2017 ) NoStop

    2017 arXiv

  55. [63]

    author author D. P. \ Kingma \ and\ author J. Ba ,\ title title Adam: A method for stochastic optimization , \ @noop journal journal arXiv preprint arXiv:1412.6980 \ ( year 2014 ) NoStop

    2014 arXiv

  56. [64]

    Heitzig , author T

    author author J. Heitzig , author T. Kittel , author J. F. \ Donges , \ and\ author N. Molkenthin ,\ title title Topology of sustainable management of dynamical systems with desirable states: from defining planetary boundaries to safe operating spaces in the earth system. \ @n...

    2016

  57. [65]

    author author M. T. \ Spaan ,\ title title Partially observable markov decision processes , \ in\ @noop booktitle Reinforcement Learning \ ( publisher Springer ,\ year 2012 )\ pp.\ pages 387--414 NoStop

    2012

  58. [66]

    Szegedy , author W

    author author C. Szegedy , author W. Zaremba , author I. Sutskever , author J. Bruna , author D. Erhan , author I. Goodfellow , \ and\ author R. Fergus ,\ title title Intriguing properties of neural networks , \ @noop journal journal arXiv preprint arXiv:1312.6199 \ ( year 201...

    2013 arXiv

  59. [67]

    Papernot , author P

    author author N. Papernot , author P. McDaniel , author S. Jha , author M. Fredrikson , author Z. B. \ Celik , \ and\ author A. Swami ,\ title title The limitations of deep learning in adversarial settings , \ in\ @noop booktitle 2016 IEEE European Symposium on Security and Pr...

    2016

  60. [68]

    Behzadan \ and\ author A

    author author V. Behzadan \ and\ author A. Munir ,\ title title Vulnerability of deep reinforcement learning to policy induction attacks , \ in\ @noop booktitle International Conference on Machine Learning and Data Mining in Pattern Recognition \ ( organization Springer ,\ yea...

    2017

  61. [69]

    Behzadan \ and\ author A

    author author V. Behzadan \ and\ author A. Munir ,\ title title Whatever does not kill deep reinforcement learning, makes it stronger , \ @noop journal journal arXiv preprint arXiv:1712.09344 \ ( year 2017 b ) NoStop

    2017 arXiv

  62. [70]

    Huang , author N

    author author S. Huang , author N. Papernot , author I. Goodfellow , author Y. Duan , \ and\ author P. Abbeel ,\ title title Adversarial attacks on neural network policies , \ @noop journal journal arXiv preprint arXiv:1702.02284 \ ( year 2017 ) NoStop

    2017 arXiv

  63. [71]

    Mirowski , author R

    author author P. Mirowski , author R. Pascanu , author F. Viola , author H. Soyer , author A. J. \ Ballard , author A. Banino , author M. Denil , author R. Goroshin , author L. Sifre , author K. Kavukcuoglu , et al. ,\ title title Learning to navigate in complex environments ,...

    2016 arXiv

  64. [72]

    author author W. D. \ Nordhaus ,\ @noop title Estimates of the social cost of carbon: background and results from the rice-2011 model , \ type Tech. Rep. \ ( institution National Bureau of Economic Research ,\ year 2011 ) NoStop

    2011

  65. [73]

    Stern \ and\ author N

    author author N. Stern \ and\ author N. H. \ Stern ,\ @noop title The economics of climate change: the Stern review \ ( publisher cambridge University press ,\ year 2007 ) NoStop

    2007

  66. [74]

    Wiedermann , author J

    author author M. Wiedermann , author J. F. \ Donges , author J. Heitzig , author W. Lucht , \ and\ author J. Kurths ,\ title title Macroscopic description of complex adaptive networks coevolving with dynamic node states , \ @noop journal journal Physical Review E \ volume 91 ,...

    2015

  67. [75]

    Barfuss , author J

    author author W. Barfuss , author J. F. \ Donges , author M. Wiedermann , \ and\ author W. Lucht ,\ title title Sustainable use of renewable resources in a stylized social--ecological network model under heterogeneous resource distribution , \ @noop journal journal Earth Syste...

    2017

  68. [76]

    Sitch , author B

    author author S. Sitch , author B. Smith , author I. C. \ Prentice , author A. Arneth , author A. Bondeau , author W. Cramer , author J. O. \ Kaplan , author S. Levis , author W. Lucht , author M. T. \ Sykes , et al. ,\ title title Evaluation of ecosystem dynamics, plant geogr...

    2003

  69. [77]

    author author J. Z. \ Leibo , author V. Zambaldi , author M. Lanctot , author J. Marecki , \ and\ author T. Graepel ,\ title title Multi-agent reinforcement learning in sequential social dilemmas , \ in\ @noop booktitle Proceedings of the 16th Conference on Autonomous Agents a...

    2017

  70. [78]

    Perolat , author J

    author author J. Perolat , author J. Z. \ Leibo , author V. Zambaldi , author C. Beattie , author K. Tuyls , \ and\ author T. Graepel ,\ title title A multi-agent reinforcement learning model of common-pool resource appropriation , \ in\ @noop booktitle Advances in Neural Info...

    2017

  71. [79]

    Heitzig , author K

    author author J. Heitzig , author K. Lessmann , \ and\ author Y. Zou ,\ title title Self-enforcing strategies to deter free-riding in the climate change mitigation game and other repeated public good games , \ @noop journal journal Proceedings of the National Academy of Scienc...

    2011

  72. [80]

    Nagabandi , author G

    author author A. Nagabandi , author G. Kahn , author R. S. \ Fearing , \ and\ author S. Levine ,\ title title Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning , \ in\ @noop booktitle 2018 IEEE International Conference on Robotics ...

    2018

  73. [81]

    Pong , author S

    author author V. Pong , author S. Gu , author M. Dalal , \ and\ author S. Levine ,\ title title Temporal difference models: Model-free deep rl for model-based control , \ @noop journal journal arXiv preprint arXiv:1802.09081 \ ( year 2018 ) NoStop

    2018 arXiv

  74. [82]

    Amodei , author C

    author author D. Amodei , author C. Olah , author J. Steinhardt , author P. Christiano , author J. Schulman , \ and\ author D. Man \'e ,\ title title Concrete problems in ai safety , \ @noop journal journal arXiv preprint arXiv:1606.06565 \ ( year 2016 ) NoStop

    2016 arXiv

  75. [83]

    author author T. M. \ Lenton , author H. Held , author E. Kriegler , author J. W. \ Hall , author W. Lucht , author S. Rahmstorf , \ and\ author H. J. \ Schellnhuber ,\ title title Tipping elements in the earth s climate system , \ 10.1073/pnas.0705414105 journal journal Proce...

    doi

  76. [84]

    author author H. J. \ Schellnhuber ,\ title title Tipping elements in the earth system , \ 10.1073/pnas.0911106106 journal journal Proceedings of the National Academy of Sciences \ volume 106 ,\ pages 20561--20563 ( year 2009 ) ,\ http://arxiv.org/abs/https://www.pnas.org/cont...

    doi

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.