Pith. sign in

REVIEW 3 major objections 5 minor 84 references

Trained distributional RL agents report risk trade-offs that are mostly training artifacts, not real environment stochasticity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 04:25 UTC pith:43GLTXCL

load-bearing objection Solid empirical audit: distributional RL risk claims are mostly training artifacts, not real aleatoric structure, and the harness is careful enough to trust. the 3 major comments →

arxiv 2607.11607 v1 pith:43GLTXCL submitted 2026-07-13 cs.AI cs.LGstat.ML

Auditing the Risk Claims of Distributional Reinforcement Learning

Hari Prasad This is my paper
classification cs.AI cs.LGstat.ML
keywords distributional reinforcement learningrisk-sensitive controlWasserstein distanceCVaRreturn distributionsstochastic dominanceaudit methodologyMinAtar
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Distributional reinforcement learning agents learn full return distributions that practitioners now treat as real risk signals for interpretability, risk-sensitive control, and safety monitoring. This paper asks whether those risk claims are true. It screens states with a decision-relevant metric that isolates genuine risk trade-offs, obtains ground truth by restarting the environment from state snapshots and rolling out thousands of futures, and tests claims with a statistical harness that prevents the audit itself from manufacturing false results. Across QR-DQN, C51, and IQN on MinAtar, 40–95% of the strongest claimed trade-offs are refuted at 95% confidence, essentially none are confirmable, and claim placement is statistically indistinguishable from truth-blind chance. The artifact forms early, is seed-idiosyncratic, uncorrelated with score, and reappears at full Atari scale. Positive controls recover real risk when it exists, showing the method measures the agents. Acting on the heads’ own risk advice can help, do nothing, or be significantly worse than chance, with no signal telling a user which regime they are in.

Core claim

For the QR-DQN, C51, and IQN agents tested, the learned “risk” in return distributions is a training artifact rather than environment stochasticity: 40–95% of the strongest claimed risk trade-offs between top actions are statistically refuted against snapshot-restart Monte Carlo ground truth, claim placement is indistinguishable from truth-blind, and essentially no claim among hundreds of audited states is confirmable.

What carries the argument

The excess Wasserstein gap Δx(s) between the agent’s top two actions by mean (1-Wasserstein distance minus absolute mean gap), which is zero exactly when one action first-order stochastically dominates the other and positive precisely when the agent claims a genuine risk trade-off. Paired with snapshot-restart Monte Carlo ground truth and a harness of permutation nulls, bootstrap refutation, and FDR control.

Load-bearing premise

That restarting the environment from deep-copied state snapshots, reseeding independently, forcing an action, and then following the greedy policy yields unbiased samples of the true return law, with the aliasing index adequately bounding residual partial-observability effects.

What would settle it

A trained QR-DQN, C51, or IQN agent on MinAtar or sticky-action Atari whose top-2% excess-Wasserstein claims are mostly confirmed by the same snapshot-restart Monte Carlo audit at FDR 0.1, with clearly positive learned–true correlation rather than near-zero correlation and high refutation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Face-value use of distributional heads for risk-sensitive policies or safety monitoring is least reliable exactly at the states practitioners would flag first.
  • Interpretability visualizations of return distributions require ground-truth validation before they can be trusted as beliefs about risk.
  • Training for risk, ensembling, and monotone recalibration do not restore confirmable top claims; recalibration “passes” only by collapsing claims to noise.
  • Future calibrated distributional methods should be held to the concrete standard of confirming their top claims under this audit.
  • Released toolkit and documented pitfalls (RNG cloning, empirical Wasserstein noise floor) give a reusable benchmark for restart-based evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The result strengthens the view that distributional RL can improve mean estimation and representation learning even when the shape of the return distribution is wrong.
  • Environments richer in genuine action-conditional aleatoric risk may raise confirmation rates without removing the winner’s-curse refutation of top spurious claims.
  • Any method that bootstraps distributional targets from a changing policy under function approximation may inherit the same artifact, including continuous-control and actor-critic variants not tested here.
  • Safety monitors that threshold head spread or CVaR may be reacting to network noise rather than environment hazard at the most-flagged states.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper asks whether the risk claims of trained distributional RL agents (QR-DQN, C51, IQN) are true when read at face value for interpretability, CVaR control, or safety monitoring. It introduces a decision-relevant screening statistic—the excess Wasserstein gap Δ_x(s) between the top-two mean-ranked actions, equal to the mass of first-order stochastic dominance violation—pairs it with snapshot-restart Monte Carlo ground truth, and wraps both in a statistical harness (permutation nulls, bootstrap refutation, FDR control, aliasing index). Across 33 MinAtar runs the strongest claims are largely refuted (40–95% top-strata), claim placement is indistinguishable from a truth-blind null, and essentially no claim is confirmable; the pattern is structural (early-formed, score-uncorrelated, seed-idiosyncratic), replicates under three projections and on a pretrained ALE QR-DQN, and is not repaired by CVaR training, ensembling, or recalibration. Positive controls with known trade-offs recover 96–100% of real claims (r≈0.89–0.92). The authors release the toolkit and document two silent restart-evaluation pitfalls.

Significance. If the measurement holds, the result is consequential for a large and growing use of distributional heads: face-value risk reads are least reliable precisely where practitioners rank them highest, and several published risk-sensitive and uncertainty pipelines rest on an untested assumption. Strengths that raise the contribution above a negative result include (i) a decision-level metric with a clean FOSD characterization (Prop. 1), (ii) an external MC answer key plus positive controls that invert the verdict at every detectable effect size, (iii) a truth-blind permutation null that pins the winner’s-curse profile, (iv) replications across three projection families and a near-SOTA ALE agent, and (v) open code, checkpoints, and raw returns that make the audit a reusable standard. The work is empirical rather than theoretical, but it supplies the missing decision-level counterpart to known fixed-point and aleatoric/epistemic warnings and is therefore a natural benchmark for future calibrated methods.

major comments (3)
  1. [Abstract / ALE audit] Abstract and §“C51, IQN, and a Pretrained Atari Agent”: the phrase “appears unchanged at full-Atari scale, with every top Breakout claim … refuted” is stronger than the body allows. The zoo agents were trained without sticky actions, so claimed spread cannot reflect environment stochasticity by construction; the paper correctly labels this a consistency check / artifact-size measurement (Table 3, Limitations). Soften the abstract and the corresponding claim in the introduction so that the ALE result is not read as an independent replication under matched training dynamics.
  2. [Decision-level evaluation / Table 2] §“Does Acting on the Heads Matter?” and Table 2: the decision-level evaluation is important but rests on small numbers of decidable states (n=14–35 per cell). The Seaquest anti-predictive result is statistically significant by binomial test, yet the Breakout “genuinely informative” claim and the cross-algorithm pattern would be more robust with bootstrap CIs on safer-pick rates and regret, or with an explicit power discussion. Without that, the environment-dependence conclusion is suggestive rather than fully quantified.
  3. [Audit Methodology / aliasing] Supplement §3.3 (aliasing index): the index is built from per-snapshot means and is therefore a coarse control that pure shape-only aliasing would evade. Low-alias strata already show equal or higher refutation, which is reassuring, but the main text should state more explicitly that residual partial-observability remains a possible confounder for the absolute confirmation rate (not for the refutation/anti-correlation core). A short sensitivity note or a second aliasing diagnostic would close the gap.
minor comments (5)
  1. [Two Pitfalls / Figure 1] Figure 1 caption and P2 discussion: the n^{-1/2} floor is classical; a one-line pointer to the Fournier–Guillin constant (or a brief derivation in the supplement) would help readers who want to re-use the null construction at other sample sizes.
  2. [Experiments / Tables 1–3] Table 1 vs. Table 3: C51 claims are quantized to the atom grid (bin width 0.8). The restriction of cross-algorithm comparison to claims above that floor is stated once; repeating it in the table notes would prevent mis-reading of the lower Asterix refutation rate.
  3. [Preliminaries / Prop. 1] Prop. 1 is proved in the supplement; a one-sentence sketch in the main text (the identity ∫|h|−|∫h|=2 min(∫h+,∫h−)) would make the FOSD link self-contained for readers who skip the supplement.
  4. [Limitations] Supplement §9 (sticky-action ALE training): the honest report of undertrained agents is welcome; a single sentence in the main Limitations stating that the released trainer makes the missing sticky-trained ALE audit a compute rather than method question would improve completeness.
  5. [References] Minor typography: “Garc ´ıa”, “B ¨ohmer”, “Fern ´andez” appear with stray spaces in the reference list; clean for camera-ready.

Circularity Check

0 steps flagged

No significant circularity: external MC ground truth, positive-control answer keys, and truth-blind permutation nulls make the refutation rates independent measurements, not constructions from the learned head.

full rationale

The paper's central claim is an empirical audit of trained distributional heads (QR-DQN, C51, IQN), not a first-principles derivation. Screening uses the excess Wasserstein gap Δx between top-two actions (Prop. 1: Δx equals the mass of FOSD violation), which is a closed-form functional of the head's quantiles; ground truth is obtained by snapshot-restart Monte Carlo rollouts that are independent of the learned head (independently reseeded clones, forced first action then greedy continuation). Confirmation and refutation tests are calibrated against per-state permutation nulls and bootstrap percentiles of the MC excess, with FDR control—none of these quantities is fitted from the claim magnitudes being tested. The truth-blind null permutes learned claim magnitudes across audited states while preserving rank strata, yielding top-strata refutation statistically indistinguishable from the observed rate (p=0.22), so placement is not forced by the metric definition. Positive controls (RISKYGRID / faint variant) supply known trade-off magnitudes by construction and invert the verdict (96–100% confirmation, r=0.89–0.92) under the identical harness, demonstrating that the audit does not manufacture zero confirmations by construction. Recalibration and distillation experiments further separate missing information from scale error rather than redefining the target. Citations (Bellemare, Dabney, Rowland, Clements, etc.) are external theory and prior methods; there is no self-citation load-bearing chain, no uniqueness theorem imported from the same author, and no fitted parameter renamed as a prediction. The result is a measurement of trained agents against external rollouts, not a reduction of outputs to inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central empirical claim rests on standard optimal-transport identities, the usual distributional-RL semantics of Z^\pi, and the operational assumption that independently reseeded snapshot restarts recover that law. No free parameters are fitted to produce the refutation rates; thresholds (FDR 0.1, n=2000) are conventional and validated by positive controls. The excess Wasserstein gap is a derived screening statistic, not an ontological invention.

free parameters (3)
  • FDR level = 0.10
    Benjamini-Hochberg threshold set to 0.10 (pooled per game); conventional choice, not fitted to maximize refutations.
  • MC rollouts per state-action = 2000
    n=2000 (MinAtar) / 1000 (ALE) chosen for power against the empirical Wasserstein noise floor; validated by positive-control recovery down to g=0.05.
  • rank strata boundaries = top 0.5/2/5/10/25/50/100 %
    Seven fixed percentiles (top 0.5%, 0.5-2%, …) used for stratified sampling; not optimized post-hoc.
axioms (4)
  • standard math W_1(F,G)=\int|F-G| and E[G]-E[F]=\int(F-G), so excess Wasserstein equals twice the smaller signed mass of F-G (Prop. 1).
    Classical one-dimensional optimal transport (Villani 2009); used to justify that \Delta_x>0 isolates first-order stochastic-dominance violations.
  • domain assumption Snapshot-restart Monte Carlo with independent reseeding recovers samples from the true return law Z^\pi under the evaluation policy.
    Core ground-truth construction (Audit Methodology §3.2); the two documented pitfalls (RNG cloning, empirical-Wasserstein floor) are the main threats to this axiom.
  • domain assumption Downstream users (CVaR policies, interpretability visualizations, safety monitors) consume the trained head as if it recovers Z^\pi.
    Stated explicitly in Introduction and Scope; the audit targets the consumed object, not an idealized Bellman fixed point.
  • domain assumption MinAtar sticky actions (p=0.1) and random spawns constitute the sole aleatoric noise sources matching the semantics of Z^\pi.
    Environment definition used for both training and MC ground truth.
invented entities (1)
  • excess Wasserstein gap \Delta_x(s) independent evidence
    purpose: Screen states for decision-relevant risk trade-offs (neither action FOSD-dominates the other) while ignoring shared location errors.
    Defined from existing W_1 and means; Prop. 1 supplies the dominance characterization. Not an ontological postulate.

pith-pipeline@v1.1.0-grok45 · 26870 in / 3055 out tokens · 38147 ms · 2026-07-14T04:25:33.162649+00:00 · methodology

0 comments
read the original abstract

Distributional reinforcement learning agents learn full return distributions that are increasingly read at face value: for interpretability, risk-sensitive control, and safety monitoring. We ask a question theory anticipates but that has not been measured directly: are the risk claims of a trained distributional agent true? Our audit combines a decision-relevant screening metric (the excess Wasserstein gap between the top two actions, which equals the mass by which first-order stochastic dominance is violated), ground truth from snapshot-restart Monte Carlo, and a statistical harness (permutation nulls, bootstrap refutation, FDR control) without which the audit itself manufactures false conclusions. Across QR-DQN, C51, and IQN on MinAtar (33 runs), 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence, the placement of the strongest claims is statistically indistinguishable from truth-blind, and essentially no claim is confirmable: for these agents, the learned "risk" reflects a training artifact rather than environment stochasticity. The artifact is structural (fully formed early in training, uncorrelated with final score, idiosyncratic to each seed) and appears unchanged at full-Atari scale, with every top Breakout claim of a pretrained near-state-of-the-art QR-DQN refuted. Positive controls of known magnitude confirm 96-100% of real claims (correlation 0.89-0.92): the reading measures the agents, not the audit. Acting on the heads' CVaR advice at their most-flagged states ranges from beneficial to significantly worse than chance. Neither training for risk nor ensembling removes the artifact, and recalibration passes the audit only by nullifying the claims: the head is uninformative, not merely miscalibrated. We release the toolkit and document two silent pitfalls that produced convincing but wrong audits of our own.

Figures

Figures reproduced from arXiv: 2607.11607 by Hari Prasad.

Figure 1
Figure 1. Figure 1: Pitfall P2, measured. Empirical excess between [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training curves for all 33 MinAtar agents (rolling mean over 200 episodes). Final scores (mean of last 100 episodes, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Refutation rate by rank stratum (QR-DQN, pooled [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Learned excess vs. ground-truth (MC) excess for all 735 QR-DQN audited states (color = rank stratum, 0 = top of [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Positive controls. Left: on RISKYGRID, where trade-offs exist by construction, the learned ranking tracks ground truth (pooled r = 0.92; per-seed 0.96/0.92/0.89; the head compresses magnitudes but orders them correctly); compare the flat clouds of [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Refutation rate by rank stratum for C51 (left) and [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Full audits of intermediate checkpoints (Breakout [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 1
Figure 1. Figure 1: Pitfall P2, measured. Empirical excess between two [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training curves for all 33 MinAtar agents (rolling mean over 200 episodes). The audit is within [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left: refutation rate by rank stratum (QR-DQN, pooled over 5 seeds, Wilson 95% intervals, FDR [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Learned excess vs. ground-truth (MC) excess for all 735 QR-DQN audited states (color = rank [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Observed top-strata refutation (red) sits inside the truth-blind null band (blue), [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Refutation by stratum for C51 (left) and IQN (right), pooled over 3 seeds. Same monotone decline [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: RISKYGRID positive control: confirmation rate by stratum (left) and CVaR-choice agreement with ground truth (right). 10.2 RISKYGRIDFAINT (detection-floor trade-offs) The faint variant straddles the audit’s detection floor [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Positive-control summary (main-text figure, reproduced). Left: [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: CVaR-choice agreement with ground truth, QR-DQN / C51 / IQN. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Top-strata refutation is flat across training while score doubles. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Learned vs. MC excess for the CVaR-greedy agent (top) and the ensemble barycenter (bottom), pooled over 3 seeds per game (color = stratum). The clouds remain flat: the interventions do not move the top claims onto the diagonal. tail location of maximum divergence, quantile skew, per-action means) and fit a leak-free logistic probe to predict the per-state refutation label, under leave-one-game-out and lea… view at source ↗
Figure 12
Figure 12. Figure 12: Audit-passing recalibration: magnitude retained (solid) and real trade-offs still asserted (hatched). [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Audited specimens: environment snapshot and both actions’ return CDFs, learned (solid) vs. [PITH_FULL_IMAGE:figures/full_fig_p025_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

84 extracted references · 8 linked inside Pith

  1. [1]

    and Dabney, Will and Munos, R

    Bellemare, Marc G. and Dabney, Will and Munos, R. A Distributional Perspective on Reinforcement Learning , booktitle =

  2. [2]

    and Munos, R

    Dabney, Will and Rowland, Mark and Bellemare, Marc G. and Munos, R. Distributional Reinforcement Learning with Quantile Regression , booktitle =

  3. [3]

    Implicit Quantile Networks for Distributional Reinforcement Learning , booktitle =

    Dabney, Will and Ostrovski, Georg and Silver, David and Munos, R. Implicit Quantile Networks for Distributional Reinforcement Learning , booktitle =

  4. [4]

    Advances in Neural Information Processing Systems 32 , year =

    Yang, Derek and Zhao, Li and Lin, Zichuan and Qin, Tao and Bian, Jiang and Liu, Tie-Yan , title =. Advances in Neural Information Processing Systems 32 , year =

  5. [5]

    Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence , year =

    Morimura, Tetsuro and Sugiyama, Masashi and Kashima, Hisashi and Hachiya, Hirotaka and Tanaka, Toshiyuki , title =. Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence , year =

  6. [6]

    Proceedings of the 32nd AAAI Conference on Artificial Intelligence , year =

    Hessel, Matteo and Modayil, Joseph and van Hasselt, Hado and Schaul, Tom and Ostrovski, Georg and Dabney, Will and Horgan, Dan and Piot, Bilal and Azar, Mohammad and Silver, David , title =. Proceedings of the 32nd AAAI Conference on Artificial Intelligence , year =

  7. [7]

    and Dabney, Will and Rowland, Mark , title =

    Bellemare, Marc G. and Dabney, Will and Rowland, Mark , title =

  8. [8]

    and Dabney, Will and Munos, R

    Rowland, Mark and Bellemare, Marc G. and Dabney, Will and Munos, R. An Analysis of Categorical Distributional Reinforcement Learning , booktitle =

  9. [9]

    Statistics and Samples in Distributional Reinforcement Learning , booktitle =

    Rowland, Mark and Dadashi, Robert and Kumar, Saurabh and Munos, R. Statistics and Samples in Distributional Reinforcement Learning , booktitle =

  10. [10]

    An Analysis of Quantile Temporal-Difference Learning , journal =

    Rowland, Mark and Munos, R. An Analysis of Quantile Temporal-Difference Learning , journal =

  11. [11]

    The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation , booktitle =

    Rowland, Mark and Tang, Yunhao and Lyle, Clare and Munos, R. The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation , booktitle =

  12. [12]

    and Castro, Pablo Samuel , title =

    Lyle, Clare and Bellemare, Marc G. and Castro, Pablo Samuel , title =. Proceedings of the 33rd AAAI Conference on Artificial Intelligence , year =

  13. [13]

    Proceedings of the 35th AAAI Conference on Artificial Intelligence , year =

    Nguyen-Tang, Thanh and Gupta, Sunil and Venkatesh, Svetha , title =. Proceedings of the 35th AAAI Conference on Artificial Intelligence , year =

  14. [14]

    Proceedings of the 36th International Conference on Machine Learning , year =

    Mavrin, Borislav and Yao, Hengshuai and Kong, Linglong and Wu, Kaiwen and Yu, Yaoliang , title =. Proceedings of the 36th International Conference on Machine Learning , year =

  15. [15]

    A Distributional Code for Value in Dopamine-Based Reinforcement Learning , journal =

    Dabney, Will and Kurth-Nelson, Zeb and Uchida, Naoshige and Starkweather, Clara Kwon and Hassabis, Demis and Munos, R. A Distributional Code for Value in Dopamine-Based Reinforcement Learning , journal =

  16. [16]

    and Danihelka, Ivo and Dabney, Will and Mohamed, Shakir and Lakshminarayanan, Balaji and Hoyer, Stephan and Munos, R

    Bellemare, Marc G. and Danihelka, Ivo and Dabney, Will and Mohamed, Shakir and Lakshminarayanan, Balaji and Hoyer, Stephan and Munos, R. The Cram. arXiv preprint arXiv:1705.10743 , year =

    Pith/arXiv arXiv

  17. [17]

    , title =

    Sobel, Matthew J. , title =. Journal of Applied Probability , volume =

  18. [18]

    Mathematical Finance , volume =

    Artzner, Philippe and Delbaen, Freddy and Eber, Jean-Marc and Heath, David , title =. Mathematical Finance , volume =

  19. [19]

    Tyrrell and Uryasev, Stanislav , title =

    Rockafellar, R. Tyrrell and Uryasev, Stanislav , title =. Journal of Risk , volume =

  20. [20]

    Proceedings of the 29th AAAI Conference on Artificial Intelligence , year =

    Tamar, Aviv and Glassner, Yonatan and Mannor, Shie , title =. Proceedings of the 29th AAAI Conference on Artificial Intelligence , year =

  21. [21]

    Advances in Neural Information Processing Systems 28 , year =

    Chow, Yinlam and Tamar, Aviv and Mannor, Shie and Pavone, Marco , title =. Advances in Neural Information Processing Systems 28 , year =

  22. [22]

    Proceedings of the 34th AAAI Conference on Artificial Intelligence , year =

    Keramati, Ramtin and Dann, Christoph and Tamkin, Alex and Brunskill, Emma , title =. Proceedings of the 34th AAAI Conference on Artificial Intelligence , year =

  23. [23]

    Proceedings of the 3rd Conference on Robot Learning , year =

    Tang, Yichuan Charlie and Zhang, Jian and Salakhutdinov, Ruslan , title =. Proceedings of the 3rd Conference on Robot Learning , year =

  24. [24]

    Advances in Neural Information Processing Systems 35 , year =

    Lim, Shiau Hong and Malik, Ilyas , title =. Advances in Neural Information Processing Systems 35 , year =

  25. [25]

    Advances in Neural Information Processing Systems 35 , year =

    Greenberg, Ido and Chow, Yinlam and Ghavamzadeh, Mohammad and Mannor, Shie , title =. Advances in Neural Information Processing Systems 35 , year =

  26. [26]

    and Van Delft, Bastien and Robaglia, Beno

    Clements, William R. and Van Delft, Bastien and Robaglia, Beno. Estimating Risk and Uncertainty in Deep Reinforcement Learning , journal =

  27. [28]

    Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning , journal =

    Charpentier, Bertrand and Senanayake, Ransalu and Kochenderfer, Mykel and G. Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning , journal =

  28. [29]

    Zanger, Moritz A. and B. Diverse Projection Ensembles for Distributional Reinforcement Learning , booktitle =

  29. [31]

    Advances in Neural Information Processing Systems 38 , year =

    Gan, Feichen and Lu, Youcun and Zhang, Yingying and Liu, Yukun , title =. Advances in Neural Information Processing Systems 38 , year =

  30. [32]

    Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence , year =

    Eriksson, Hannes and Basu, Debabrota and Alibeigi, Mina and Dimitrakakis, Christos , title =. Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence , year =

  31. [33]

    Advances in Neural Information Processing Systems 29 , year =

    Osband, Ian and Blundell, Charles and Pritzel, Alexander and Van Roy, Benjamin , title =. Advances in Neural Information Processing Systems 29 , year =

  32. [34]

    Proceedings of the 33rd International Conference on Machine Learning , year =

    Gal, Yarin and Ghahramani, Zoubin , title =. Proceedings of the 33rd International Conference on Machine Learning , year =

  33. [35]

    Advances in Neural Information Processing Systems 30 , year =

    Kendall, Alex and Gal, Yarin , title =. Advances in Neural Information Processing Systems 30 , year =

  34. [36]

    Advances in Neural Information Processing Systems 30 , year =

    Lakshminarayanan, Balaji and Pritzel, Alexander and Blundell, Charles , title =. Advances in Neural Information Processing Systems 30 , year =

  35. [37]

    Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-Sensitive Learning , booktitle =

    Depeweg, Stefan and Hern. Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-Sensitive Learning , booktitle =

  36. [38]

    , title =

    Guo, Chuan and Pleiss, Geoff and Sun, Yu and Weinberger, Kilian Q. , title =. Proceedings of the 34th International Conference on Machine Learning , year =

  37. [39]

    Proceedings of the 40th International Conference on Machine Learning , year =

    Wu, Runzhe and Uehara, Masatoshi and Sun, Wen , title =. Proceedings of the 40th International Conference on Machine Learning , year =

  38. [40]

    arXiv preprint arXiv:2309.17262 , year =

    Zhang, Liangyu and Peng, Yang and Yang, Wenhao and Zhang, Zhihua , title =. arXiv preprint arXiv:2309.17262 , year =

    arXiv

  39. [41]

    and Abate, Alessandro and Johansson, Karl Henrik , title =

    Wang, Zifan and Gao, Yulong and Wang, Siyi and Zavlanos, Michael M. and Abate, Alessandro and Johansson, Karl Henrik , title =. Learning for Dynamics and Control Conference , year =

  40. [42]

    and Bellemare, Marc G

    Machado, Marlos C. and Bellemare, Marc G. and Talvitie, Erik and Veness, Joel and Hausknecht, Matthew and Bowling, Michael , title =. Journal of Artificial Intelligence Research , volume =

  41. [43]

    and Naddaf, Yavar and Veness, Joel and Bowling, Michael , title =

    Bellemare, Marc G. and Naddaf, Yavar and Veness, Joel and Bowling, Michael , title =. Journal of Artificial Intelligence Research , volume =

  42. [45]

    and Veness, Joel and Bellemare, Marc G

    Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Rusu, Andrei A. and Veness, Joel and Bellemare, Marc G. and Graves, Alex and Riedmiller, Martin and Fidjeland, Andreas K. and Ostrovski, Georg and Petersen, Stig and Beattie, Charles and Sadik, Amir and Antonoglou, Ioannis and King, Helen and Kumaran, Dharshan and Wierstra, Daan and Legg, Shane ...

  43. [46]

    Proceedings of the 32nd AAAI Conference on Artificial Intelligence , year =

    Henderson, Peter and Islam, Riashat and Bachman, Philip and Pineau, Joelle and Precup, Doina and Meger, David , title =. Proceedings of the 32nd AAAI Conference on Artificial Intelligence , year =

  44. [47]

    and Bellemare, Marc G

    Agarwal, Rishabh and Schwarzer, Max and Castro, Pablo Samuel and Courville, Aaron C. and Bellemare, Marc G. , title =. Advances in Neural Information Processing Systems 34 , year =

  45. [48]

    How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments , journal =

    Colas, C. How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments , journal =

  46. [49]

    Journal of the Royal Statistical Society: Series B , volume =

    Benjamini, Yoav and Hochberg, Yosef , title =. Journal of the Royal Statistical Society: Series B , volume =

  47. [50]

    , title =

    Efron, Bradley and Tibshirani, Robert J. , title =

  48. [51]

    Probability Theory and Related Fields , volume =

    Fournier, Nicolas and Guillin, Arnaud , title =. Probability Theory and Related Fields , volume =

  49. [52]

    Bernoulli , volume =

    Weed, Jonathan and Bach, Francis , title =. Bernoulli , volume =

  50. [53]

    On Wasserstein Two-Sample Testing and Related Families of Nonparametric Tests , journal =

    Ramdas, Aaditya and Garc. On Wasserstein Two-Sample Testing and Related Families of Nonparametric Tests , journal =

  51. [54]

    Optimal Transport: Old and New , publisher =

    Villani, C. Optimal Transport: Old and New , publisher =

  52. [55]

    Proceedings of the 35th International Conference on Machine Learning , year =

    Greydanus, Samuel and Koul, Anurag and Dodge, Jonathan and Fern, Alan , title =. Proceedings of the 35th International Conference on Machine Learning , year =

  53. [56]

    Proceedings of the 8th International Conference on Learning Representations , year =

    Atrey, Akanksha and Clary, Kaleigh and Jensen, David , title =. Proceedings of the 8th International Conference on Learning Representations , year =

  54. [57]

    Concrete Problems in

    Amodei, Dario and Olah, Chris and Steinhardt, Jacob and Christiano, Paul and Schulman, John and Man. Concrete Problems in. arXiv preprint arXiv:1606.06565 , year =

    Pith/arXiv arXiv

  55. [58]

    A Comprehensive Survey on Safe Reinforcement Learning , journal =

    Garc. A Comprehensive Survey on Safe Reinforcement Learning , journal =

  56. [59]

    S.; Courville, A

    Agarwal, R.; Schwarzer, M.; Castro, P. S.; Courville, A. C.; and Bellemare, M. G. 2021. Deep Reinforcement Learning at the Edge of the Statistical Precipice. In Advances in Neural Information Processing Systems 34

    2021

  57. [60]

    Artzner, P.; Delbaen, F.; Eber, J.-M.; and Heath, D. 1999. Coherent Measures of Risk. Mathematical Finance, 9(3): 203--228

    1999

  58. [61]

    Atrey, A.; Clary, K.; and Jensen, D. 2020. Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning. In Proceedings of the 8th International Conference on Learning Representations

    2020

  59. [62]

    G.; Dabney, W.; and Munos, R

    Bellemare, M. G.; Dabney, W.; and Munos, R. 2017. A Distributional Perspective on Reinforcement Learning. In Proceedings of the 34th International Conference on Machine Learning

    2017

  60. [63]

    G.; Dabney, W.; and Rowland, M

    Bellemare, M. G.; Dabney, W.; and Rowland, M. 2023. Distributional Reinforcement Learning. MIT Press

    2023

  61. [64]

    Benjamini, Y.; and Hochberg, Y. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B, 57(1): 289--300

    1995

  62. [65]

    Charpentier, B.; Senanayake, R.; Kochenderfer, M.; and G \"u nnemann, S. 2022. Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning. arXiv preprint arXiv:2206.01558

    Pith/arXiv arXiv 2022

  63. [66]

    R.; Van Delft, B.; Robaglia, B.-M.; Slaoui, R

    Clements, W. R.; Van Delft, B.; Robaglia, B.-M.; Slaoui, R. B.; and Toth, S. 2019. Estimating Risk and Uncertainty in Deep Reinforcement Learning. arXiv preprint arXiv:1905.09638

    Pith/arXiv arXiv 2019

  64. [67]

    Dabney, W.; Ostrovski, G.; Silver, D.; and Munos, R. 2018 a . Implicit Quantile Networks for Distributional Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning

    2018

  65. [68]

    G.; and Munos, R

    Dabney, W.; Rowland, M.; Bellemare, M. G.; and Munos, R. 2018 b . Distributional Reinforcement Learning with Quantile Regression. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence

    2018

  66. [69]

    Eriksson, H.; Basu, D.; Alibeigi, M.; and Dimitrakakis, C. 2022. SENTINEL : Taming Uncertainty with Ensemble Based Distributional Reinforcement Learning. In Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence

    2022

  67. [70]

    Fournier, N.; and Guillin, A. 2015. On the Rate of Convergence in Wasserstein Distance of the Empirical Measure. Probability Theory and Related Fields, 162: 707--738

    2015

  68. [71]

    Gan, F.; Lu, Y.; Zhang, Y.; and Liu, Y. 2025. Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation. In Advances in Neural Information Processing Systems 38

    2025

  69. [72]

    Garc \'i a, J.; and Fern \'a ndez, F. 2015. A Comprehensive Survey on Safe Reinforcement Learning. Journal of Machine Learning Research, 16: 1437--1480

    2015

  70. [73]

    Greydanus, S.; Koul, A.; Dodge, J.; and Fern, A. 2018. Visualizing and Understanding Atari Agents. In Proceedings of the 35th International Conference on Machine Learning

    2018

  71. [74]

    Hoel, C.-J.; Wolff, K.; and Laine, L. 2021. Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving. arXiv preprint arXiv:2105.10266

    Pith/arXiv arXiv 2021

  72. [75]

    Keramati, R.; Dann, C.; Tamkin, A.; and Brunskill, E. 2020. Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy. In Proceedings of the 34th AAAI Conference on Artificial Intelligence

    2020

  73. [76]

    H.; and Malik, I

    Lim, S. H.; and Malik, I. 2022. Distributional Reinforcement Learning for Risk-Sensitive Policies. In Advances in Neural Information Processing Systems 35

    2022

  74. [77]

    G.; and Castro, P

    Lyle, C.; Bellemare, M. G.; and Castro, P. S. 2019. A Comparative Analysis of Expected and Distributional Reinforcement Learning. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence

    2019

  75. [78]

    C.; Bellemare, M

    Machado, M. C.; Bellemare, M. G.; Talvitie, E.; Veness, J.; Hausknecht, M.; and Bowling, M. 2018. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. Journal of Artificial Intelligence Research, 61: 523--562

    2018

  76. [79]

    T.; and Uryasev, S

    Rockafellar, R. T.; and Uryasev, S. 2000. Optimization of Conditional Value-at-Risk. Journal of Risk, 2(3): 21--41

    2000

  77. [80]

    G.; Dabney, W.; Munos, R.; and Teh, Y

    Rowland, M.; Bellemare, M. G.; Dabney, W.; Munos, R.; and Teh, Y. W. 2018. An Analysis of Categorical Distributional Reinforcement Learning. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics

    2018

  78. [81]

    G.; Tang, Y.; Ostrovski, G.; Harutyunyan, A.; Tuyls, K.; Bellemare, M

    Rowland, M.; Munos, R.; Azar, M. G.; Tang, Y.; Ostrovski, G.; Harutyunyan, A.; Tuyls, K.; Bellemare, M. G.; and Dabney, W. 2023 a . An Analysis of Quantile Temporal-Difference Learning. arXiv preprint arXiv:2301.04462

    Pith/arXiv arXiv 2023

  79. [82]

    G.; and Dabney, W

    Rowland, M.; Tang, Y.; Lyle, C.; Munos, R.; Bellemare, M. G.; and Dabney, W. 2023 b . The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation. In Proceedings of the 40th International Conference on Machine Learning

    2023

  80. [83]

    C.; Erricolo, D.; Tulabandhula, T.; and Trivedi, A

    Stutts, A. C.; Erricolo, D.; Tulabandhula, T.; and Trivedi, A. R. 2024. Echoes of Socratic Doubt: Embracing Uncertainty in Calibrated Evidential Reinforcement Learning. arXiv preprint arXiv:2402.07107

    Pith/arXiv arXiv 2024

Showing first 80 references.