Pith. sign in

REVIEW 4 major objections 5 minor 59 references

Generalization in Monitored Markov Decision Processes (Mon-MDPs)

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that coupling a learned reward model with function approximation lets reinforcement-learning agents generalize from states with visible rewards to states where rewards are hidden, achieving near-optimal behavior in…

desk verdict A first empirical look at function approximation for Mon-MDPs with a real finding about overgeneralization, but the main 'near-optimal' claim outruns the evidence. read the letter →

arxiv 2505.08988 v1 pith:KW6TJEV4 submitted 2025-05-13 cs.AI

Montaser Mohammedalamen , Michael Bowling This is my paper · ORCID
classification cs.AI
keywords monitoredMarkovdecisionprocessesunobservablerewardsfunctionapproximationrewardmodelgeneralizationovergeneralizationrobustpolicyoptimizationepistemicuncertaintyreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether reinforcement learning can act sensibly when rewards are only sometimes observable, such as a plant-watering robot that receives feedback only while the homeowner is watching. It extends the Mon-MDP framework, previously limited to small tabular environments, to high-dimensional state spaces by pairing a learned reward model with function approximation. The central claim is that the reward model can generalize from monitored states to unmonitored states, yielding near-optimal policies in environments that were formally unsolvable in tabular Mon-MDPs. The paper also shows the same generalization can go wrong, and that a cautious policy optimization based on reward uncertainty reduces the harm.

What carries the argument

The load-bearing object is the learned reward model $\hat{R}(s^E, a^E; \theta)$, a convolutional neural network trained only on transitions whose proxy reward equals the true environment reward. It feeds predicted rewards into a deep Q-network that also receives the monitor state, so generalization across the monitored and unmonitored boundary is carried by the CNN's feature-sharing, which maps visually similar states to similar reward predictions. To counter overgeneralization, the paper adapts a cautious optimizer: an ensemble of 500 reward models quantifies epistemic uncertainty, and $k$-of-$N$ counterfactual regret minimization optimizes the average of the $k$ worst models, interpolating between risk-neutral and highly risk-averse behavior.

What would settle it

Run the half-room experiment while gradually increasing the visual divergence between Zone 1 and Zone 2 (changing colors, shapes, or adding objects) and measure the gap between the reward-model policy's return and an oracle that knows true rewards in Zone 2; if the gap stays near zero even for large divergence, the generalization claim is stronger than the similarity assumption predicts, while a growing gap would confirm that the claim is bounded by feature transfer.

Watch

Extended reading notes

Core claim

The paper demonstrates that a reward model trained with function approximation on states where rewards are observable can predict rewards in states where rewards are not observable, allowing the agent to learn near-optimal policies even in Mon-MDPs that are formally unsolvable in the tabular setting because the function approximator's inductive bias effectively selects one reward function among indistinguishable ones. In a half-room environment with monitoring only on the left, the agent waters plants in the unmonitored right zone at a comparable rate to the monitored zone. The paper also shows that this same generalization mechanism overreaches: an agent that only observed rewards for standard plants waters cacti in the unmonitored zone despite their distinct representation and negative true reward. To mitigate this, the paper adapts a cautious policy optimization method using an ensemble of reward models and $k$-of-$N$ counterfactual regret minimization, which reduces watering of novel plants while preserving watering of familiar ones.

Load-bearing premise

The whole transfer result rests on unmonitored states looking and behaving enough like monitored states that the features learned in the monitored zone stay predictive in the unmonitored zone.

Editorial extensions

If this is right

  • In the binary-monitor plant-watering environment, the reward-model policy converges to near-optimal behavior and stops requesting monitoring, while treating hidden rewards as zero or discarding them leaves the agent stuck with suboptimal monitoring behavior.
  • In the half-room environment, the agent waters plants in the unmonitored zone about as often as in the monitored zone, which the paper takes as evidence that reward knowledge transfers across the monitored boundary even though the environment is formally unsolvable in the tabular setting.
  • Overgeneralization is real: an agent that only ever observed rewards for standard plants also waters cacti in the unmonitored zone, despite their distinct appearance and negative true reward.
  • Cautious policies built from a reward-model ensemble and $k$-of-$N$ CFR reduce watering of novel plants by up to five times while keeping watering of familiar plants, so robustness tuning can partially offset overgeneralization.
  • The paper's results position function approximation as the ingredient that lets Mon-MDPs move beyond tabular environments, and its appendix points to plasticity loss as a separate obstacle for deep Mon-MDP training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the visual-similarity assumption is the real carrier of generalization, then the formal 'unsolvable' status of a Mon-MDP is representation-dependent: a tabular agent cannot distinguish reward functions, but a CNN-based reward model can, by effectively selecting one reward function through its inductive bias.
  • A practical design suggested by the results is to treat reward-model uncertainty as a signal for when to request monitoring or abstain: the cacti episode is exactly a case where the model is confident but wrong, so confidence alone may be insufficient.
  • The cautious $k$-of-$N$ method likely trades off sensitivity: it will also reduce watering of genuinely rewarding novel plants that share few features with familiar plants, so deployment would need a separate mechanism to discover new reward sources rather than simply avoiding them.
  • The half-room and botanical-garden experiments together imply a testable scaling relation between visual divergence of the unmonitored zone and performance loss; measuring that curve would place a quantitative bound on when the reward-model approach applies.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies monitored Markov decision processes (Mon-MDPs) beyond the tabular case. It trains a convolutional reward model on observed rewards and a DQN on predicted rewards, and compares against baselines that treat unobserved rewards as zero or discard those transitions. Experiments cover a binary monitor, a half-room with one monitored half, a plant/cactus environment, and a botanical garden where an ensemble reward model with k-of-N CFR is used for cautious policy optimization. The paper claims that function approximation plus a learned reward model generalizes from monitored to unmonitored states, achieves near-optimal policies in some formally unsolvable Mon-MDPs, and that cautious policy optimization mitigates the overgeneralization exhibited by non-robust function approximation.

Significance. The question is significant: whether function approximation can overcome the tabular indistinguishability barrier in Mon-MDPs is an important step toward practical applications. The paper has concrete strengths: experiments use multiple seeds, hyperparameter tuning is reported, an explicit failure case is studied, and code is included with the submission. The cautious-policy experiments are a useful first step. However, the headline claims on near-optimality and on overcoming formal unsolvability are not backed by the reported measurements, and an appendix result appears to contradict the main-text convergence claim. If the near-optimality evidence is supplied, the contribution would be valuable.

major comments (4)
  1. [Section 4.2, Figure 4a] The claim that the reward model achieves near-optimal policies in the half-room environment is not measured against any optimality baseline. The environment reward structure is fully known: watering dry or partially dry plants gives +1, watering wet plants gives -1, watering empty cells gives -0.2, and there is no monitoring penalty in this environment. An optimal return or a strong upper bound is therefore computable. The figure only shows that discounted rewards in Zone 1 and Zone 2 are comparable, which demonstrates consistency across zones but not proximity to optimal. This is load-bearing because the abstract and Contribution 2 assert near-optimality in formally unsolvable Mon-MDPs.
  2. [Section 4.2 and Section 2.2] The formal definition of an unsolvable Mon-MDP is that no policy is optimal for all indistinguishable reward functions. The half-room experiment does not test this definition: it fixes one reward function for Zone 2 and shows that the CNN propagates Zone 1 rewards to Zone 2. The claim that FA enables agents to navigate some unsolvable environments conflates successful inductive guessing on one reward function with overcoming the formal barrier. The paper should either include an adversarial indistinguishability check, varying the Zone 2 reward functions and reporting regret against the optimal policy for each variant, or explicitly reframe the contribution as showing that FA can generalize on some instances of unsolvable Mon-MDPs.
  3. [Appendix A.4, Figure 8, and Section 4.1, Figure 3a] Appendix A.4 states that in the binary environment, after 6M timesteps the agent's performance deteriorates and it starts requesting monitoring again, as shown in Figure 8. The main text in Section 4.1 and Figure 3a shows training to 10M timesteps with stable high discounted reward and claims that the reward model empirically converges to the optimal policy. These two presentations are inconsistent. If performance collapses with longer training, the near-optimal claim is only a stopping-time artifact, and the manuscript must report the exact training duration used in the main curves and explain the discrepancy.
  4. [Section 4.4, Table 1] The cautious-policy result is partly by construction: k-of-N CFR optimizes against the worst k of N reward models, so a policy trained this way is expected to reduce actions in states where the ensemble is uncertain. The interesting finding is that robust policies maintain or increase watering of familiar plants while reducing novel-plant watering, but the main text reports only ratios relative to the reward model. Please present the absolute watering frequencies in the main text (Table 5 in the appendix already contains them) and include the ensemble-mean baseline without worst-case selection (the 10-of-10 result in the appendix) so that the effect of ensembling can be separated from the effect of worst-case robustness.
minor comments (5)
  1. [Abstract] The phrase 'cautious police optimization' should be 'cautious policy optimization'.
  2. [Section 3] The sentence 'the success of reward model generalization depends critically on the: i) state representation' contains a misplaced colon before the list; rephrase for clarity.
  3. [Section 4.3 and Figure 4b] The environment is called 'plant-cactus' in the text and 'plants & cacti' in the figure caption; use one consistent name.
  4. [Appendix A.1, Table 4] The table reports area under the training curve for hyperparameter tuning, but the main text never defines what a 'training curve' is for the half-room environment; please add a one-sentence definition.
  5. [Section 4.4] The description of the novel-plant representations and the reporting in Table 1 are clear, but the sentence in Section 4.4 about novel plants being 'sampled from {[0,0,1], ...}' is not followed by the distribution used for the experiments; please state whether each novel plant type is sampled uniformly at the start of each episode.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the core generalization and overgeneralization results are empirical, and the self-citations are background rather than load-bearing reductions.

full rationale

The paper's central claims are empirical demonstrations rather than derivations. The reward model is trained on monitored-zone data and evaluated on unmonitored-zone behavior; the half-room result (Fig. 4a) is a genuine measurement of transfer, not an identity with its training input. The cautious-policy result in Sec. 4.4 is unsurprising because k-of-N CFR optimizes against the worst k of N reward models, so reducing watering of high-uncertainty novel plants follows from the objective's construction; however, the paper presents this as a behavioral evaluation of an adopted algorithm, not as a first-principles prediction derived from the objective, so it is not circular in the sense of a fitted parameter being renamed as a prediction. The 'near-optimal' language is not backed by an explicit optimality baseline, and the 'unsolvable' label is imported from Parisi et al. 2024b, which includes two of the present authors; but the cited definition is a formal characterization of indistinguishability, and it is not used to force the empirical outcome. No equation in the paper reduces to its own inputs, no fitted value is relabeled as a predictive finding, and no uniqueness theorem is invoked by self-citation to preclude alternatives. The paper's main vulnerabilities are evidentiary: the near-optimal claim lacks a measured optimality gap, and the cacti experiment shows the failure mode of the same generalization mechanism. These are correctness and validation concerns, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The paper relies on standard RL assumptions plus a feature-similarity inductive bias for the reward model. The counting of tuned hyperparameters, environment constants, and the robustness-level choice reflects the empirical, design-dependent nature of the claims.

free parameters (4)
  • Dryness rate = 0.05 per timestep
    Chosen for the plant-watering simulator rather than fitted to external data; all experimental conclusions depend on this stochastic drying process.
  • Monitoring penalty rM = -0.2
    Hand-set cost for requesting monitoring; it shapes the optimal policy in the binary environment and affects the learning signal.
  • Learning rates eta_R and eta_Q = 1e-4, selected from 0.0005, 0.001, 0.005
    Tuned per Table 4 by area under the training curve; the reported results use the selected values.
  • Robustness levels k-of-N = 5-of-10 and 1-of-10
    Chosen to represent mid-level and high robustness; the cautious-policy results are defined relative to these choices.
assumptions (4)
  • domain assumption Reward model trained on monitored samples generalizes to unmonitored states through shared visual features.
    Invoked in Section 4.2 to interpret half-room success; it is the main inductive-bias assumption and fails for cacti in Section 4.3.
  • domain assumption The environment reward is a deterministic function of the agent's egocentric state-action representation.
    Assumed implicitly by the CNN reward model in Section 3; no unobserved confounders are modeled.
  • domain assumption The half-room environment is unsolvable under the Mon-MDP definition from Parisi et al. (2024b).
    Invoked in Section 4.2 to frame the result as solving a formally unsolvable case; the definition concerns indistinguishability over reward functions, not FA inductive bias.
  • ad hoc to paper Novel plants in the botanical garden have unknown but potentially negative rewards, so reduced watering is treated as safer.
    Section 4.4 evaluates caution by watering frequency; the true rewards for novel plant types are never specified, so the safety claim is inferred from behavior rather than ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generalization in Monitored Markov Decision Processes (Mon-MDPs)." pith.science (2026). https://pith.science/paper/KW6TJEV4

@misc{pith2026250508988,
  author       = {Pith},
  title        = {Pith review of: Generalization in Monitored Markov Decision Processes (Mon-MDPs)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KW6TJEV4}},
  note         = {Machine review of arXiv:2505.08988}
}
read the original abstract

Reinforcement learning (RL) typically models the interaction between the agent and environment as a Markov decision process (MDP), where the rewards that guide the agent's behavior are always observable. However, in many real-world scenarios, rewards are not always observable, which can be modeled as a monitored Markov decision process (Mon-MDP). Prior work on Mon-MDPs have been limited to simple, tabular cases, restricting their applicability to real-world problems. This work explores Mon-MDPs using function approximation (FA) and investigates the challenges involved. We show that combining function approximation with a learned reward model enables agents to generalize from monitored states with observable rewards, to unmonitored environment states with unobservable rewards. Therefore, we demonstrate that such generalization with a reward model achieves near-optimal policies in environments formally defined as unsolvable. However, we identify a critical limitation of such function approximation, where agents incorrectly extrapolate rewards due to overgeneralization, resulting in undesirable behaviors. To mitigate overgeneralization, we propose a cautious police optimization method leveraging reward uncertainty. This work serves as a step towards bridging this gap between Mon-MDP theory and real-world applications.

Figures

Figures reproduced from arXiv: 2505.08988 by the authors.

Figure 1
Figure 1. MDP and Monitored-MDP Diagrams. for each MDP Q∗(s, a) = P st+1 P(st+1|st, at) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Frames from each environment. Reward model = 0 Ignore 0 1 2 3 4 5 6 7 8 9 10 Training Timesteps (×10 6 ) 20 15 10 5 0 5 10 Discounted Joint Reward (a) Training curve. 0 1 2 3 4 5 6 7 8 9 10 Training Timesteps (×10 6 ) 0.0 0.2 0.4 0.6 0.8 1.0 Monitor action frequency (%) (b) Frequency of monitor action during training [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Reward model, “ignore”, and ⊥ = 0 performance in the binary environment. The bold line represents the mean over 10 seeds; the shaded area is a 95% interval confidence. choose from along with the six environment actions, aM ∈ {ask to be monitored, not ask to be monitored}. Therefore, if the agent asks to be monitored, it observes the true environment reward rˆ E = r E and receives a monitoring reward rM = −0.2 on tha… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Results for the half-room and plants & cacti environments. The bold line represents the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Exploration ϵ decay rate tuning in the binary and half-room environments. The bold line represents the mean over 10 seeds; the shaded area is a 95% interval confidence [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Reward model, “ignore”, and ⊥ = 0 training curves in the binary and half-room environ￾ments. The bold line represents the mean over 10 seeds; the shaded area is a 95% interval confidence. Reward model = 0 Ignore 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Training Timesteps (×10 6…
Figure 7
Figure 7. Figure 7: Reward model, “ignore”, and ⊥ = 0 training curves in the binary and half-room environ￾ments. The bold line represents the mean over 10 seeds, and each line represents a seed. A.3 Detailed results for the Botanical garden environment The Botanical Garden environment ser…
Figure 8
Figure 8. Figure 8: Preliminary evidence for the loss of plasticity in the binary environment. The bold line [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Preliminary evidence for the loss of plasticity in the half-room environment. The bold line [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 51 canonical work pages

  1. [1]

    Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. Conference on Lifelong Learning Agents, pp.\ 620--636, 2023

    work page 2023

  2. [2]

    str \"o m, K. J. Optimal control of M arkov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10: 0 174--205, 1965

    work page 1965

  3. [3]

    G., Naddaf, Y., Veness, J., and Bowling, M

    Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The A rcade L earning E nvironment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 0 253--279, 2013

    work page 2013

  4. [4]

    G., Candido, S., Castro, P

    Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., and Wang, Z. Autonomous navigation of stratospheric balloons using reinforcement learning. N ature , 588 0 (7836): 0 77--82, 2020

    work page 2020

  5. [5]

    and Kalaba, R

    Bellman, R. and Kalaba, R. Dynamic programming and statistical communication theory. Proceedings of the National Academy of Sciences, 43 0 (8): 0 749--751, 1957

    work page 1957

  6. [6]

    Binmore, K. G. Game theory: a very short introduction, volume 173. Oxford University Press, 2007

    work page 2007

  7. [7]

    and Moore, A

    Boyan, J. and Moore, A. Generalization in reinforcement learning: Safely approximating the value function. Advances in Neural Information Processing Systems, 7, 1994

    work page 1994

  8. [8]

    and Bowling, M

    Chen, K. and Bowling, M. Tractable objectives for robust policy optimization. Advances in Neural Information Processing Systems, 25: 0 2069--2077, 2012

    work page 2012

Show all 59 references
  1. [9]

    B., Abeyruwan, S., Graesser, L., Iscen, A., Amor, H

    D'Ambrosio, D. B., Abeyruwan, S., Graesser, L., Iscen, A., Amor, H. B., Bewley, A., Reed, B. J., Reymann, K., Takayama, L., Tassa, Y., et al. Achieving human level competitive robot table tennis. arXiv preprint arXiv:2408.03906, 2024

    2024 arXiv

  2. [10]

    Deisenroth, M. P. and Rasmussen, C. E. Pilco: a model-based and data-efficient approach to policy search. International Conference on International Conference on Machine Learning, pp.\ 465–472, 2011

    2011

  3. [11]

    F., Lan, Q., Rahman, P., Mahmood, A

    Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632 0 (8026): 0 768--774, 2024

    2024

  4. [12]

    and Mahmood, A

    Elsayed, M. and Mahmood, A. R. Addressing loss of plasticity and catastrophic forgetting in continual learning. International Conference on Learning Representations, 2024

    2024

  5. [13]

    Tree-based batch mode reinforcement learning

    Ernst, D., Geurts, P., and Wehenkel, L. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005

    2005

  6. [14]

    A., Gao, J., Ryan, M

    Evans, R. A., Gao, J., Ryan, M. C., Dulac-Arnold, G., Scholz, J. K., and Hester, T. A. Optimizing data center controls using neural networks, December 5 2023. US Patent 11,836,599

    2023

  7. [15]

    G., Piot, B., Menick, J., Hessel, M., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., and Legg, S

    Fortunato, M., Azar, M. G., Piot, B., Menick, J., Hessel, M., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., and Legg, S. Noisy networks for exploration. International Conference on Learning Representations, 2018

    2018

  8. [16]

    Deep sparse rectifier neural networks

    Glorot, X., Bordes, A., and Bengio, Y. Deep sparse rectifier neural networks. International Conference on Artificial Intelligence and Statistics, 15: 0 315--323, 2011

    2011

  9. [17]

    Gordon, G. J. Stable function approximation in dynamic programming. International Conference on Machine Learning, pp.\ 261--268, 1995

    1995

  10. [18]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, pp.\ 1861--1870, 2018

    2018

  11. [19]

    Dream to C ontrol: Learning behaviors by latent imagination

    Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to C ontrol: Learning behaviors by latent imagination. International Conference on Learning Representations, 2020

    2020

  12. [20]

    Halmos, P. R. Naive set theory. Van Nostrand, 1960

    1960

  13. [21]

    Practical confidence and prediction intervals

    Heskes, T. Practical confidence and prediction intervals. Advances in Neural Information Processing Systems, 9, 1996

    1996

  14. [22]

    K., Shah, H., White, M., Miahi, E., Machado, M

    Janjua, M. K., Shah, H., White, M., Miahi, E., Machado, M. C., and White, A. GVFs in the real world: making predictions online for water treatment. Machine Learning, 113 0 (8): 0 5151--5181, 2024

    2024

  15. [23]

    Monotonic robust policy optimization with model discrepancy

    Jiang, Y., Li, C., Dai, W., Zou, J., and Xiong, H. Monotonic robust policy optimization with model discrepancy. International Conference on Machine Learning, 139: 0 4951--4960, 2021

    2021

  16. [24]

    E., and Bowling, M

    Kazemipour, S., Parisi, S., Taylor, M. E., and Bowling, M. Model-based exploration in monitored markov decision processes. arXiv preprint arXiv:2502.16772, 2025

    2025

  17. [25]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. International Conference for Learning Representations, 2015

    2015

  18. [26]

    and Tsitsiklis, J

    Konda, V. and Tsitsiklis, J. Actor-critic algorithms. Advances in Neural Information Processing Systems, 12, 1999

    1999

  19. [27]

    Learning robust policy against disturbance in transition dynamics via state-conservative policy optimization

    Kuang, Y., Lu, M., Wang, J., Zhou, Q., Li, B., and Li, H. Learning robust policy against disturbance in transition dynamics via state-conservative policy optimization. AAAI Conference on Artificial Intelligence, 36: 0 7247--7254, 2022

    2022

  20. [28]

    and Sutton, R

    Kuvayev, L. and Sutton, R. S. Model-based reinforcement learning with an approximate, learned model. In Yale workshop on Adaptive and Learning Systems, pp.\ 101--105, 1996

    1996

  21. [29]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, pp.\ 6405--6416, 2017

    2017

  22. [30]

    Batch reinforcement learning

    Lange, S., Gabel, T., and Riedmiller, M. Batch reinforcement learning. In Reinforcement learning: State-of-the-art, pp.\ 45--73. Springer, 2012

    2012

  23. [31]

    Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning

    Lee, K., Laskin, M., Srinivas, A., and Abbeel, P. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. International Conference on Machine Learning, pp.\ 6131--6141, 2021

    2021

  24. [32]

    and Koltun, V

    Levine, S. and Koltun, V. Guided policy search. International Conference on Machine Learning, 28: 0 1--9, 2013

    2013

  25. [33]

    and Van Roy, B

    Lu, X. and Van Roy, B. Ensemble sampling. Advances in Neural Information Processing Systems, pp.\ 3260--3268, 2017

    2017

  26. [34]

    Playing A tari with deep reinforcement learning

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmille, M. Playing A tari with deep reinforcement learning. Advances in Neural Information Processing Systems, Deep Learning Workshop, 2013

    2013

  27. [35]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. N ature , 518 0 (7540): 0 529--533, 2015

    2015

  28. [36]

    Learning to be cautious

    Mohammedalamen, M., Morrill, D., Sieusahai, A., Satsangi, Y., and Bowling, M. Learning to be cautious. arXiv preprint arXiv:2110.15907, 2021

    2021

  29. [37]

    Deepstack: E xpert-level artificial intelligence in heads-up no-limit poker

    Morav c \' k, M., Schmid, M., Burch, N., Lis \`y , V., Morrill, D., Bard, N., Davis, T., Waugh, K., Johanson, M., and Bowling, M. Deepstack: E xpert-level artificial intelligence in heads-up no-limit poker. Science, 356 0 (6337): 0 508--513, 2017

    2017

  30. [38]

    Deep reinforcement learning with plasticity injection

    Nikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., and Barreto, A. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36: 0 37142--37159, 2023

    2023

  31. [39]

    J., and Wen, Z

    Osband, I., Van Roy, B., Russo, D. J., and Wen, Z. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20 0 (124): 0 1--62, 2019

    2019

  32. [40]

    M., Dwaracherla, V., Ibrahimi, M., Lu, X., and Van Roy, B

    Osband, I., Wen, Z., Asghari, S. M., Dwaracherla, V., Ibrahimi, M., Lu, X., and Van Roy, B. Epistemic neural networks. Advances in Neural Information Processing Systems, 36: 0 2795--2823, 2023

    2023

  33. [41]

    Papavassiliou, V. A. and Russell, S. Convergence of reinforcement learning with general function approximators. International Joint Conference on Artificial Intelligence, 99: 0 748--755, 1999

    1999

  34. [42]

    Beyond optimism: Exploration with partially observable rewards

    Parisi, S., Kazemipour, A., and Bowling, M. Beyond optimism: Exploration with partially observable rewards. Advances in Neural Information Processing Systems, 37: 0 65415--65444, 2024 a

    2024

  35. [43]

    E., and Bowling, M

    Parisi, S., Mohammedalamen, M., Kazemipour, A., Taylor, M. E., and Bowling, M. Monitored M arkov decision processes. International Conference on Autonomous Agents and Multiagent Systems, pp.\ 1549--1557, 2024 b

    2024

  36. [44]

    Automatic differentiation in pytorch

    Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. Workshop on Autodiff, Neural Information Processing Systems, 2017

    2017

  37. [45]

    Bayesian neural network ensembles

    Pearce, T., Zaki, M., and Neely, A. Bayesian neural network ensembles. In Workshop on Bayesian Deep Learning, Neural Information Processing Systems, 2018

    2018

  38. [46]

    M., Dawson, M

    Pilarski, P. M., Dawson, M. R., Degris, T., Fahimi, F., Carey, J. P., and Sutton, R. S. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. IEEE International Conference on Rehabilitation Robotics, pp.\ 1--7, 2011

    2011

  39. [47]

    H., Benosman, M., and Van Baar, J

    Russel, R. H., Benosman, M., and Van Baar, J. Robust constrained-mdps: Soft-constrained robust policy optimization under model uncertainty. arXiv preprint arXiv:2010.04870, 2020

    2010 arXiv

  40. [48]

    Trust region policy optimization

    Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. International Conference on Machine Learning, 37: 0 1889--1897, 2015

    2015

  41. [49]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  42. [50]

    J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al

    Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of G o with deep neural networks and tree search. N ature , 529 0 (7587): 0 484--489, 2016

    2016

  43. [51]

    Sutton, R. S. Learning to predict by the methods of temporal differences. Machine learning, 3: 0 9--44, 1988

    1988

  44. [52]

    S., McAllester, D., Singh, S., and Mansour, Y

    Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999

    1999

  45. [53]

    S., McAllester, D., Singh, S., and Mansour, Y

    Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, pp.\ 1057--1063, 2000

    2000

  46. [54]

    S., Szepesv \'a ri, C., Geramifard, A., and Bowling, M

    Sutton, R. S., Szepesv \'a ri, C., Geramifard, A., and Bowling, M. Dyna-style planning with linear function approximation and prioritized sweeping. Conference on Uncertainty in Artificial Intelligence, pp.\ 528--536, 2008

    2008

  47. [55]

    A comparison of some error estimates for neural network models

    Tibshirani, R. A comparison of some error estimates for neural network models. Neural Computation, 8 0 (1): 0 152--163, 1996

    1996

  48. [56]

    M., Mathieu, M., Dudzik, A., Chung, J., Choi, D

    Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in S tar C raft II using multi-agent reinforcement learning. N ature , 575 0 (7782): 0 350--354, 2019

    2019

  49. [57]

    R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T

    Wurman, P. R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T. J., Capobianco, R., Devlic, A., Eckert, F., Fuchs, F., et al. Outracing champion G ran T urismo drivers with deep reinforcement learning. N ature , 602 0 (7896): 0 223--228, 2022

    2022

  50. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.