Pith. sign in

REVIEW 2 major objections 67 references

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

T0 review · 2 major / 0 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distributional shift in RL is a change in one of five generative factors of the agent-environment loop, not a vague train-test mismatch.

desk verdict Clean conceptual taxonomy that unifies ID/OOD and non-stationarity via POMDP factors; useful framing, thin single-run evidence. read the letter →

arxiv 2606.16933 v2 pith:UQD5Q6TW submitted 2026-06-15 cs.LG cs.AI

Ardianto Wibowo , Paulo E Santos , Amer Baghdadi , Matthew Stephenson , Karl Sammut , Jean-Philippe Diguet This is my paper
classification cs.LGcs.AI
keywords distributionalshiftreinforcementlearningPOMDPcausal-origintaxonomyID/OODgeneralizationnon-stationarityadaptationmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning agents fail when the world or their own behavior stops matching the conditions under which they were trained. The paper argues that those failures are not one generic problem but mismatches that can be localized to specific pieces of the interaction: how states are distributed, how observations are generated, how the policy chooses actions, how the environment transitions, or how rewards are assigned. By rewriting classical supervised dataset shift in terms of this five-factor generative process, the authors show that the familiar labels “in-distribution / out-of-distribution” and “stationary / non-stationary” are simply different ways of describing when and where that process changes. They further distinguish agent-driven (internal) from environment-driven (external) shifts and explicit, implicit, or hybrid time boundaries. The payoff is a shared vocabulary for diagnosing robustness failures and for measuring not only whether performance drops but how hard the drop is and how quickly an agent recovers.

What carries the argument

The five-factor POMDP generative factorization p(s,o,a,s′,r)=p(s)p(o|s)π(a|o)p(s′|s,a)p(r|s,a,s′), split into external (environment) and internal (agent) factors and crossed with a shifted-time boundary that may be explicit, implicit, or hybrid.

What would settle it

Find a realistic RL failure mode whose performance collapse cannot be attributed to any change among p(s), p(o|s), π(a|o), p(s′|s,a), or p(r|s,a,s′), or show that two different single-factor interventions produce statistically indistinguishable degradation and recovery profiles under the paper’s own metrics.

Watch

Extended reading notes

Core claim

Distributional shift in reinforcement learning is fully characterized as a mismatch between the pre-boundary and post-boundary joint generative models p(o,s,a,s′,r). That mismatch must arise from a change in at least one of five causal-origin factors—p(s), p(o|s), π(a|o), p(s′|s,a), or p(r|s,a,s′)—and the same factorization unifies classical ID/OOD generalization with non-stationarity under explicit, implicit, or hybrid shifted-time boundaries.

Load-bearing premise

That every practically important distributional shift can be cleanly located in one of those five factors plus a single time boundary, and that single-run gridworld experiments are enough to show the resulting degradation patterns are distinct and diagnosable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper proposes a causal-origin taxonomy of distributional shift in RL by transferring the classical supervised dataset-shift factorization to the POMDP generative process. It decomposes a single interaction step as s ~ p(s), o ~ p(o|s), a ~ π(a|o), s' ~ p(s'|s,a), r ~ p(r|s,a,s') (Eqs. 5–10), then defines distributional shift as any pre/post mismatch in the joint generative model that originates in at least one of these five factors (Definitions 1–3). Shifts are further classified as internal (agent-driven: p(o|s), π(a|o)) or external (environment-driven: p(s), p(s'|s,a), p(r|s,a,s')), and as explicit, implicit, or hybrid according to the shifted-time boundary. The taxonomy is extended to decentralized MARL (Definition 4), illustrated with single-run DQN experiments on a 30×30 gridworld under five hand-designed perturbations (§5), and accompanied by five adaptation metrics (Δ_shift, Δ_drop, T_rec, Γ_post, A_adapt) plus a post-hoc mapping of existing methods (Table 3).

Significance. If accepted as a conceptual contribution, the paper supplies a clean, shared vocabulary that unifies ID/OOD generalization and non-stationarity under a single generative factorization. The internal/external split and the explicit/implicit/hybrid boundary distinction are useful diagnostic tools for method design and for interpreting why robustness is shift-type dependent (as suggested by Robust-Gymnasium and RL-ViGen). The evaluation metrics in §6 are a concrete, reusable addition that go beyond aggregate return. The formal core (Eqs. 5–10, Definitions 1–4) is definitional and consistent with standard POMDP theory; the main value is organizational rather than predictive. The empirical section is intentionally minimal and does not claim algorithmic novelty.

major comments (2)
  1. §5.5 and the claim that the taxonomy yields “distinct and diagnosable degradation patterns” rest on single independent runs with no error bars, no statistical tests, and a deliberately minimal 30×30 gridworld. The authors themselves note that each result is from one run. This is too thin to support a diagnostic claim; either expand to multi-seed statistics (or a second environment) or rephrase the empirical section as a purely illustrative demonstration of the definitions rather than evidence of diagnostic power.
  2. The sufficiency claim for the five-factor factorization plus a single shifted-time boundary (Definitions 1–3, §4) is asserted rather than stress-tested. Mixed or higher-order cases (e.g., a visual change that simultaneously alters p(o|s) and p(r|s,a,s'), or continuous multi-agent co-adaptation that blurs the internal/external boundary) are only briefly acknowledged in the Discussion. A short subsection clarifying the taxonomy’s scope and known incompletenesses would strengthen the central claim without requiring new theory.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the taxonomy is a definitional re-factorization of the standard POMDP generative process and does not reduce any claimed result to a fitted input or self-citation chain.

full rationale

The paper's central claim (Definitions 1–3 and the §4 taxonomy) is obtained by writing the standard POMDP one-step generative process (Eq. 5) as the joint factorization p(s,o,a,s′,r)=p(s)p(o|s)π(a|o)p(s′|s,a)p(r|s,a,s′) (Eqs. 6–10), then declaring a distributional shift whenever any factor differs across a shifted-time boundary. This is a transparent definitional re-organization of classical MDP/POMDP theory (Sutton & Barto, Kaelbling et al.) together with the classical dataset-shift factorization transferred from supervised learning; it does not derive a numerical quantity from a parameter fitted inside the paper and later re-presented as a prediction. The evaluation metrics (Δshift, Δdrop, Trec, Γpost, Aadapt) are likewise pure definitions applied to hand-designed single-run gridworld curves; they contain no fitted parameters that are then called predictions. Mapping of prior methods (Table 3) is post-hoc classification, not a load-bearing derivation. There are no self-citations that supply uniqueness theorems, ansätze, or other premises required by the taxonomy. The derivation chain is therefore self-contained against external benchmarks and exhibits none of the six circularity patterns.

Assumptions & free parameters 3 free parameters · 4 assumptions · 3 invented entities

The paper rests almost entirely on standard MDP/POMDP generative assumptions and the classical dataset-shift factorization transferred from supervised learning. No free parameters are fitted to produce the taxonomy itself; the only free choices are experimental design decisions (grid size, FOV roll amount, quantization bits, window fractions for metrics). Invented entities are limited to the named taxonomy categories and the five evaluation metrics, which are definitional rather than physical postulates.

free parameters (3)
  • pre/post-shift window fractions and recovery threshold η
    User-defined evaluation parameters (W_pre = 10 % of shift period, W_post = 70 %, η = 0.9 in the example) that directly affect the numerical values of the five adaptation metrics; not derived from first principles.
  • quantization bits and clipping threshold for policy-induced shift
    Hand-chosen n_bits and c that control the strength of the synthetic internal policy distortion; chosen for illustration rather than measured from hardware.
  • FOV roll amount n and deadline T_max
    Hand-designed perturbation magnitudes used to instantiate observation-mapping and reward-mechanism shifts; not fitted but still free experimental knobs.
assumptions (4)
  • domain assumption Standard POMDP generative factorization p(s,o,a,s′,r) = p(s)p(o|s)π(a|o)p(s′|s,a)p(r|s,a,s′)
    Taken as the complete causal structure of agent–environment interaction (§4.1, Eq. 5–6); any shift outside these five factors is outside the taxonomy by construction.
  • domain assumption Classical supervised dataset-shift factorization P(x,y)=P(y|x)P(x) can be transferred by grouping X=(s,o), Y=(a,s′,r)
    The conceptual bridge that justifies calling the five-factor mismatch “distributional shift” (§4.1, Eqs. 7–9).
  • domain assumption In fully decentralized MARL, other agents’ evolving policies appear only as external changes to the effective transition and reward kernels
    Definition 4; standard non-stationarity argument restated inside the taxonomy.
  • ad hoc to paper A single shifted-time boundary (explicit, implicit, or hybrid) is sufficient to locate the regime change
    Introduced in §4.2; continuous or multi-point drifts are not given separate formal status.
invented entities (3)
  • Internal vs external distributional shift categories
    purpose: Partition the five generative factors by whether the agent or the environment controls them
    Definitional labels derived from the source of control; no independent physical existence claimed.
  • Explicit / implicit / hybrid shifted-time boundary
    purpose: Classify how the before/after regime change is operationally realized
    Organizational device for unifying ID/OOD with non-stationarity; not an observable physical object.
  • Five adaptation metrics (Δ_shift, Δ_drop, T_rec, Γ_post, A_adapt)
    purpose: Quantify immediate impact, worst degradation, recovery speed, quality, and efficiency after a shift
    New evaluation quantities defined from user-chosen windows; their usefulness is claimed but not independently validated beyond one illustrative table.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning." pith.science (2026). https://pith.science/paper/UQD5Q6TW

@misc{pith2026260616933,
  author       = {Pith},
  title        = {Pith review of: A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQD5Q6TW}},
  note         = {Machine review of arXiv:2606.16933}
}
read the original abstract

Reinforcement learning (RL) systems often degrade when operating conditions differ from those previously encountered, reflecting distributional shifts in the underlying data-generating process. Such shifts may occur between training and evaluation, as in In-Distribution (ID) and Out-of-Distribution (OOD) generalization, or within non-stationary settings where environment dynamics evolve over time. However, the formal relationship between these views remains unclear, and existing work mainly focuses on mitigation rather than the causal origin of shift within the agent-environment interaction. This work develops a unified causal-origin taxonomy that characterizes sources of distributional shift in RL and relates ID/OOD generalization to non-stationary settings. We transfer the classical dataset-shift principle from supervised learning to RL by reformulating distributional shift in terms of the generative interaction process. Using a Partially Observable Markov Decision Process (POMDP), we decompose the interaction into structural components, including the state distribution, observation process, policy, reward, and transition dynamics, together with the shifted-time boundary. The proposed taxonomy distinguishes internal (agent-driven) and external (environment-driven) distributional shifts. The shifted-time boundary perspective further characterizes explicit, implicit, and hybrid shifts. This formulation unifies ID/OOD generalization and non-stationarity as structured changes in the underlying process. We also introduce an evaluation framework for measuring shift impact and adaptation through performance degradation and recovery metrics. By grounding distributional shift in the causal-origin structure of RL, this work supports systematic analysis of robustness under distributional shift.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 13 canonical work pages

  1. [1]

    Distributionally Adaptive Meta Reinforcement Learning

    Ajay, A. et al. (2022). “Distributionally Adaptive Meta Reinforcement Learning”. In:Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS)

    2022

  2. [2]

    CROP: Towards Distributional-Shift Robust Reinforcement Learning Using Compact Reshaped Observation Processing

    Altmann, P. et al. (2023). “CROP: Towards Distributional-Shift Robust Reinforcement Learning Using Compact Reshaped Observation Processing”. In:Proceedings of the 32th International Joint Conference on Artificial Intelligence (IJCAI). Macao, SAR China, pp. 3414–3422. isbn: 978-1-956792-03-4.doi: 10.24963/ijcai.2023/380.url: https://www.ijcai.org/ proceedi...

    work page doi:10.24963/ijcai.2023/380.url: 2023

  3. [3]

    Smooth Q-Learning: An Algorithm for Independent Learners in Stochastic Cooperative Markov Games

    Amhraoui, E. and T. Masrour (2023). “Smooth Q-Learning: An Algorithm for Independent Learners in Stochastic Cooperative Markov Games”. In:Journal of Intelligent & Robotic Systems108.4, p. 65.issn: 0921-0296, 1573-0409.doi: 10.1007/s10846-023-01917-z.url: https://link.springer.com/10.1007/s10846-023-01917-z(visited on 09/15/2024)

    work page doi:10.1007/s10846-023-01917-z.url: 2023

  4. [4]

    Optimal Control of Markov Processes with Incomplete State Information

    Amodei, D. et al. (2016).Concrete Problems in AI Safety. arXiv:1606.06565.doi:10.48550/ arXiv.1606.06565.url:http://arxiv.org/abs/1606.06565(visited on 02/17/2026). Åström, K. (1965). “Optimal Control of Markov Processes with Incomplete State Information”. In: Journal of Mathematical Analysis and Applications10.1, pp. 174–205.issn: 0022247X.doi: 10.1016/0...

    arXiv doi:10.1016/0022- 2016

  5. [5]

    Athanasios, P. and S. U. Pillai (2002).Probability, Random Variables, and Stochastic Processes. 4th. McGraw-Hill. 34

    2002

  6. [6]

    Beck, J. et al. (2024).A Survey of Meta-Reinforcement Learning. arXiv:2301.08028.doi: 10 . 48550 / arXiv . 2301 . 08028.url: http : / / arxiv . org / abs / 2301 . 08028(visited on 04/15/2025)

    arXiv 2024

  7. [7]

    Foresight Distribution Adjustment for Off-policy Reinforcement Learning

    Chen, R. (2024). “Foresight Distribution Adjustment for Off-policy Reinforcement Learning”. In:Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS)

    2024

  8. [8]

    Quantifying Generalization in Reinforcement Learning

    Cobbe, K. et al. (2019). “Quantifying Generalization in Reinforcement Learning”. In:Proceedings of the 36th International Conference on Machine Learning (ICML). Da Silva, B. C. et al. (2006). “Improving Reinforcement Learning with Context Detection”. In: Proceedings of the 5th International Joint Conference on Autonomous Agents and Multiagent Systems (AAM...

    work page doi:10.1145/1160633.1160779 2019

Show all 67 references
  1. [9]

    Györfi, and G

    Devroye, L., L. Györfi, and G. Lugosi (2008).A Probabilistic Theory of Pattern Recognition. 3rd. New York, NY: Springer.isbn: 978-0-387-94618-4

    2008

  2. [10]

    (1990).Stochastic Process

    Doob, J. (1990).Stochastic Process. New York: Wiley

    1990

  3. [11]

    Multi-Agent Reinforcement Learning Clustering Algorithm based on Silhouette Coefficient

    Du, P., F. Li, and J. Shao (2024). “Multi-Agent Reinforcement Learning Clustering Algorithm based on Silhouette Coefficient”. In:Neurocomputing596, p. 127901.issn: 09252312.doi: 10.1016/j.neucom.2024.127901 .url: https://linkinghub.elsevier.com/retrieve/ pii/S0925231224006726(...

    2024 doi

  4. [12]

    Duan, Y. et al. (2016).RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv:1611.02779.doi: 10.48550/arXiv.1611.02779.url: http://arxiv.org/abs/1611. 02779(visited on 11/17/2025)

    2016 doi arXiv

  5. [13]

    Challenges of Real-world Reinforcement Learning: Definitions, Benchmarks and Analysis

    Dulac-Arnold, G. et al. (2021). “Challenges of Real-world Reinforcement Learning: Definitions, Benchmarks and Analysis”. In:Machine Learning110.9, pp. 2419–2468.issn: 0885-6125, 1573-0565.doi: 10.1007/s10994- 021- 05961- 4.url: https://link.springer.com/10. 1007/s10994-021-059...

    2021 doi

  6. [14]

    Echchahed, A. and P. S. Castro (2025).A Survey of State Representation Learning for Deep Reinforcement Learning. en. arXiv:2506.17518.doi: 10.48550/arXiv.2506.17518 .url: http://arxiv.org/abs/2506.17518(visited on 11/16/2025)

    2025 doi arXiv

  7. [15]

    Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learning

    Emami, P. et al. (2023). “Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learning”. In:Proceedings of the 62nd of IEEE Conference on Decision and Control (CDC). Singapore: IEEE, pp. 2372–2378.isbn: 979-8-3503-0124-3.doi:10.1109/ CDC49753 . 2023 . ...

    2023

  8. [16]

    Learned Step Size Quantization

    Esser, S. K. et al. (2020). “Learned Step Size Quantization”. In:Proceedings of the 8th International Conference on Learning Representations (ICLR)

    2020

  9. [17]

    Abbeel, and S

    Finn, C., P. Abbeel, and S. Levine (2017).Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv:1703.03400.doi: 10.48550/arXiv.1703.03400.url: http://arxiv. org/abs/1703.03400(visited on 11/17/2025)

    2017 doi arXiv

  10. [18]

    Assessing the Impact of Distribution Shift on Reinforcement Learning Performance

    Fujimoto, T. et al. (2023). “Assessing the Impact of Distribution Shift on Reinforcement Learning Performance”.In:Workshop on Regulatable Machine Learning at the 37th Conference on Neural Information Processing Systems (RegML @ NeurIPS). arXiv:2402.03590.doi:10.48550/ arXiv.24...

    2023 arXiv

  11. [19]

    Robust Gymnasium: A Unified Modular Benchmark for Robust Re- inforcement Learning

    Gu, S. et al. (2025). “Robust Gymnasium: A Unified Modular Benchmark for Robust Re- inforcement Learning”. In:Procedings of the 13th International Conference on Learning Representations (ICLR)

    2025

  12. [20]

    Domain Shifts in Reinforcement Learning: Identifying Disturbances in Environments

    Haider, T. et al. (2021). “Domain Shifts in Reinforcement Learning: Identifying Disturbances in Environments”. In:CEUR Workshop Proceedings2916. 35

    2021

  13. [21]

    Anwar, and S

    Herremans, S., A. Anwar, and S. Mercelis (2024).Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model. arXiv:2406.09976.doi:10.48550/arXiv.2406.09976. url:http://arxiv.org/abs/2406.09976(visited on 11/23/2025)

    2024 doi arXiv

  14. [22]

    Hybrid Safe Reinforcement Learning: Tackling Distribution Shift and Outliers with the Student-t’s Process

    Hickman, X., Y. Lu, and D. Prince (2025). “Hybrid Safe Reinforcement Learning: Tackling Distribution Shift and Outliers with the Student-t’s Process”. In:Neurocomputing634, p.129912.issn:09252312.doi: 10.1016/j.neucom.2025.129912.url: https://linkinghub. elsevier.com/retrieve/...

    2025 doi

  15. [23]

    Howard, R. A. (1960).Dynamic Programming and Markov Processes. The Technology Press of The Massachusetts Institute of Technology and John Wiley & Sons, Inc

    1960

  16. [24]

    Quantization and Training of Neural Networks for Efficient Integer- Arithmetic-Only Inference

    Jacob, B. et al. (2018). “Quantization and Training of Neural Networks for Efficient Integer- Arithmetic-Only Inference”. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT: IEEE, pp. 2704–2713.isbn: 978-1-5386-6420-9. doi:...

    2018 doi

  17. [25]

    I2Q: A Fully Decentralized Q-Learning Algorithm

    Jiang, J. and Z. Lu (2022). “I2Q: A Fully Decentralized Q-Learning Algorithm”. In:Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS)

    2022

  18. [26]

    New Reinforcement Learning based on Representation Transfer for Portfolio Management

    Jiang, W. et al. (2024). “New Reinforcement Learning based on Representation Transfer for Portfolio Management”. In:Knowledge-Based Systems293, p. 111697.issn: 09507051.doi: 10.1016/j.knosys.2024.111697 .url: https://linkinghub.elsevier.com/retrieve/ pii/S0950705124003320(visi...

    2024 doi

  19. [27]

    Planning and Acting in Partially Observable Stochastic Domains

    Kaelbling, L. P., M. L. Littman, and A. R. Cassandra (1998). “Planning and Acting in Partially Observable Stochastic Domains”. In:Artificial Intelligence101.1-2, pp. 99–134.issn: 00043702. doi: 10 . 1016 / S0004 - 3702(98 ) 00023 - X.url: https : / / linkinghub . elsevier . co...

    1998

  20. [28]

    Towards Continual Reinforcement Learning: A Review and Perspec- tives

    Khetarpal, K. et al. (2022). “Towards Continual Reinforcement Learning: A Review and Perspec- tives”. In:Journal of Artificial Intelligence Research75, pp. 1401–1476.issn: 1076-9757.doi: 10.1613/jair.1.13673.url: https://jair.org/index.php/jair/article/view/13673 (visited on 1...

    2022 doi

  21. [29]

    A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

    Kirk, R. et al. (2023). “A Survey of Zero-shot Generalisation in Deep Reinforcement Learning”. In: Journal of Artificial Intelligence Research76, pp. 201–264.issn: 1076-9757.doi:10.1613/ jair.1.14174.url: http://jair.org/index.php/jair/article/view/14174 (visited on 11/16/2025...

    2023

  22. [30]

    Addressing Distribution Shift in Online Reinforcement Learning with Offline Datasets

    Lee, S. et al. (2020). “Addressing Distribution Shift in Online Reinforcement Learning with Offline Datasets”. In:Offline Reinforcement Learning Workshop at Neural Information Processing Systems.url:https://neurips.cc/virtual/2020/20101

    2020

  23. [31]

    Levin, D. A. and Y. Peres (2009).Markov Chains and Mixing Times. 2nd. American Mathematical Society

    2009

  24. [32]

    Dealing with Non-Stationarity In MARL via Trust-Region Decomposition

    Li, W. et al. (2022). “Dealing with Non-Stationarity In MARL via Trust-Region Decomposition”. In:Procedings of the 10th International Conference on Learning Representations (ICLR)

    2022

  25. [33]

    Liu, J. et al. (2023).Towards Out-Of-Distribution Generalization: A Survey. arXiv:2108.13624. doi: 10.48550/arXiv.2108.13624.url: http://arxiv.org/abs/2108.13624 (visited on 11/16/2025)

    2023 doi arXiv

  26. [34]

    Szepesvári, and C

    Liu, Q., C. Szepesvári, and C. Jin (2022).Sample-Efficient Reinforcement Learning of Partially Observable Markov Games. arXiv:2206.01315

    2022 arXiv

  27. [35]

    Liu, Z. et al. (2024).A Behavior-Aware Approach for Deep Reinforcement Learning in Non- stationary Environments without Known Change Points. arXiv:2405.14214.doi:10.48550/ arXiv.2405.14214.url:http://arxiv.org/abs/2405.14214(visited on 11/17/2025)

    2024 arXiv

  28. [36]

    Learning under Concept Drift: A Review

    Lu, J. et al. (2018). “Learning under Concept Drift: A Review”. en. In:IEEE Transactions on Knowledge and Data Engineering, pp. 1–1.issn: 1041-4347, 1558-2191, 2326-3865.doi: 36 10.1109/TKDE.2018.2876857 .url: https://ieeexplore.ieee.org/document/8496795/ (visited on 10/31/2025)

    2018 doi

  29. [37]

    Luo, W. et al. (2025).Mitigating Distribution Shift in Model-based Offline RL via Shifts-aware Reward Learning. arXiv:2408.12830.doi:10.48550/arXiv.2408.12830.url: http://arxiv. org/abs/2408.12830(visited on 10/31/2025)

    2025 doi

  30. [38]

    Predictive Uncertainty Estimation via Prior Networks

    Malinin, A. and M. Gales (2018). “Predictive Uncertainty Estimation via Prior Networks”. In: Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS)

    2018

  31. [39]

    Mnih, V. et al. (2013).Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602. doi: 10.48550/arXiv.1312.5602 .url: http://arxiv.org/abs/1312.5602 (visited on 04/26/2025)

    2013 doi arXiv

  32. [40]

    Taxonomy of Machine Learning Safety: A Survey and Primer

    Mohseni, S. et al. (2023). “Taxonomy of Machine Learning Safety: A Survey and Primer”. In: ACM Computing Surveys55.8, pp. 1–38.issn: 0360-0300, 1557-7341.doi:10.1145/3551385. url:https://dl.acm.org/doi/10.1145/3551385(visited on 11/27/2025)

    2023 doi

  33. [41]

    Neural Posterior Domain Randomization

    Muratore, F. et al. (2021). “Neural Posterior Domain Randomization”. In:Procedings of the 5th Conference on Robot Learning (CoRL)

    2021

  34. [42]

    Neggatu, N. S., J. Houssineau, and G. Montana (2025).Evaluation-Time Policy Switching for Offline Reinforcement Learning. arXiv:2503.12222.doi:10.48550/arXiv.2503.12222.url: http://arxiv.org/abs/2503.12222(visited on 04/15/2025)

    2025 doi arXiv

  35. [43]

    (1997).Markov Chain

    Norris, J. (1997).Markov Chain. Cambridge University Press

    1997

  36. [44]

    Can You Trust Your Model’s Uncertainty? Evaluating Predictive UncertaintyunderDatasetShift

    Ovadia, Y. et al. (2019). “Can You Trust Your Model’s Uncertainty? Evaluating Predictive UncertaintyunderDatasetShift”.In:Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS). Packer,C.etal.(2019).Assessing Generalization in Deep Reinforcement ...

    2019 doi arXiv

  37. [45]

    A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments

    Padakandla, S. (2022). “A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments”. In:ACM Computing Surveys54.6, pp. 1–25.issn: 0360-0300, 1557-7341. doi: 10.1145/3459991 .url: https://dl.acm.org/doi/10.1145/3459991 (visited on 10/31/2025)

    2022 doi

  38. [46]

    Reinforcement Learning Algorithm for Non- Stationary Environments

    Padakandla, S., P. K. J., and S. Bhatnagar (2020). “Reinforcement Learning Algorithm for Non- Stationary Environments”. In:Applied Intelligence50.11, pp. 3590–3606.issn: 0924-669X, 1573-7497.doi: 10.1007/s10489- 020- 01758- 5.url: https://link.springer.com/10. 1007/s10489-020-...

    2020 doi

  39. [47]

    Papadopoulos, G. et al. (2025).An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks. arXiv:2502.04773.doi:10.48550/ arXiv.2502.04773.url:http://arxiv.org/abs/2502.04773(visited on 04/16/2026)

    2025 arXiv

  40. [48]

    Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks

    Papoudakis, G. and F. Christianos (2021). “Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks”. In:Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS)

    2021

  41. [49]

    Christianos, et al

    Papoudakis, G., F. Christianos, et al. (2019).Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning. arXiv:1906.04737

    2019 arXiv

  42. [50]

    Puterman, M. L. (2005).Markov Decision Process. Wiley-Interscience. Quiñonero-Candela, J. et al., eds. (2010).Dataset shift in machine learning. Neural information processing series. Cambridge, Mass: MIT Press.isbn: 978-0-262-17005-5 978-0-262-25510-3

    2005

  43. [51]

    Efficient Sim-to-Real Transfer in Rein- forcement Learning Through Domain Randomization and Domain Adaptation

    Shakerimov, A., T. Alizadeh, and H. A. Varol (2023). “Efficient Sim-to-Real Transfer in Rein- forcement Learning Through Domain Randomization and Domain Adaptation”. In:IEEE Access11, pp. 136809–136824.issn: 2169-3536.doi: 10.1109/ACCESS.2023.3339568.url: https://ieeexplore.ie...

    2023 doi

  44. [52]

    Shalev-Shwartz, S. and S. Ben-David (2014).Understanding Machine Learning: From Theory to Algorithms. 1st ed. Cambridge University Press.isbn: 978-1-107-05713-5 978-1-107-29801-9. 37 doi: 10 . 1017 / CBO9781107298019.url: https : / / www . cambridge . org / core / product / id...

    2014

  45. [53]

    Al-Shedivat, M. et al. (2018).Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments. arXiv:1710.03641.doi: 10 . 48550 / arXiv . 1710 . 03641.url: http://arxiv.org/abs/1710.03641(visited on 11/17/2025)

    2018 arXiv

  46. [54]

    CovariateShiftAdaptationbyImportanceWeightedCrossValidation

    Slaoui, R. B. et al. (2020).Robust Visual Domain Randomization for Reinforcement Learning. arXiv:1910.10537.doi: 10.48550/arXiv.1910.10537.url: http://arxiv.org/abs/1910. 10537(visited on 11/19/2025). Sugiyama,M.etal.(2007).“CovariateShiftAdaptationbyImportanceWeightedCrossVal...

    2020 doi arXiv

  47. [55]

    Sutton, R. S. and A. G. Barto (2018).Reinforcement Learning: an Introduction. Second edition. Adaptive Computation and Machine Learning Series. Cambridge, Massachusetts: The MIT Press.isbn: 978-0-262-03924-6

    2018

  48. [56]

    Handling Out-of-Distribution Data: A Survey

    Tamang, L. et al. (2025). “Handling Out-of-Distribution Data: A Survey”. In:IEEE Transactions on Knowledge and Data Engineering37.10, pp. 5948–5966.issn: 1041-4347, 1558-2191, 2326- 3865.doi: 10.1109/TKDE.2025.3592614.url: https://ieeexplore.ieee.org/document/ 11098614/(visite...

    2025 doi

  49. [57]

    DROPO: Sim-to-Real Transfer with Offline Domain Randomization

    Tiboni, G., K. Arndt, and V. Kyrki (2023). “DROPO: Sim-to-Real Transfer with Offline Domain Randomization”. In:Robotics and Autonomous Systems166, p. 104432.issn: 09218890.doi: 10.1016/j.robot.2023.104432 .url: https://linkinghub.elsevier.com/retrieve/ pii/S0921889023000714(vi...

    2023 doi

  50. [58]

    Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

    Tobin, J. et al. (2017). “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”. en. In:Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Vancouver, BC: IEEE, pp. 23–30.isbn: 978-1-5386- 2682-...

    2017 doi

  51. [59]

    An Overview of Statistical Learning Theory

    Vapnik, V. (1999). “An Overview of Statistical Learning Theory”. In:IEEE Transactions on Neural Networks10.5, pp. 988–999.issn: 10459227.doi: 10.1109/72.788640.url: http: //ieeexplore.ieee.org/document/788640/(visited on 11/01/2025)

    1999 doi

  52. [60]

    Improving Generalization in Offline Reinforcement Learning via Latent Distribution Representation Learning

    Wang, D. et al. (2025). “Improving Generalization in Offline Reinforcement Learning via Latent Distribution Representation Learning”. In:Proceedings of the 39th AAAI Conference on Artificial Intelligence

    2025

  53. [61]

    Decentralized Multi-Agent Cooperation via Adaptive Partner Modeling

    Xu, C. et al. (2024). “Decentralized Multi-Agent Cooperation via Adaptive Partner Modeling”. In:Complex & Intelligent Systems10.4, pp. 4989–5004.issn: 2199-4536, 2198-6053.doi: 10.1007/s40747-024-01421-3.url: https://link.springer.com/10.1007/s40747-024- 01421-3(visited on 09/06/2024)

    2024 doi

  54. [62]

    Meta-Reinforcement Learning Robust to Distributional Shift Via Performing Lifelong In-Context Learning

    Xu, T., Z. Li, and Q. Ren (2024). “Meta-Reinforcement Learning Robust to Distributional Shift Via Performing Lifelong In-Context Learning”. In:Proceedings of the 41st International Conference on Machine Learning (ICML)

    2024

  55. [63]

    RL-ViGen: A Reinforcement Learning Benchmark for Visual General- ization

    Yuan, Z. et al. (2023). “RL-ViGen: A Reinforcement Learning Benchmark for Visual General- ization”. In:Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)

    2023

  56. [64]

    Dynamic Belief for Decentralized Multi-Agent Cooperative Learning

    Zhai, Y. et al. (2023). “Dynamic Belief for Decentralized Multi-Agent Cooperative Learning”. In:Proceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI)

    2023

  57. [65]

    344–352.isbn: 978-1-956792-03-4.doi:10.24963/ijcai.2023/39

    Macau, SAR China, pp. 344–352.isbn: 978-1-956792-03-4.doi:10.24963/ijcai.2023/39. url:https://www.ijcai.org/proceedings/2023/39(visited on 09/06/2024)

    2023 doi

  58. [66]

    Zhang, C. et al. (2018).A Study on Overfitting in Deep Reinforcement Learning. arXiv:1804.06893. doi: 10.48550/arXiv.1804.06893.url: http://arxiv.org/abs/1804.06893 (visited on 04/14/2025)

    2018 doi arXiv

  59. [67]

    Sim-to-Real Transfer in Deep Reinforce- ment Learning for Robotics: a Survey

    Zhao, W., J. P. Queralta, and T. Westerlund (2020). “Sim-to-Real Transfer in Deep Reinforce- ment Learning for Robotics: a Survey”. In:Proceedings of the IEEE Symposium Series on 38 Computational Intelligence (SSCI). arXiv:2009.13303, pp. 737–744.doi:10.1109/SSCI47803. 2020.93...

    2020 doi arXiv

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.