Pith. sign in

REVIEW 5 major objections 6 minor 39 references

Causal-Paced Deep Reinforcement Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper tries to establish that task selection in curriculum reinforcement learning can be driven by an interaction-data proxy for causal task difference, avoiding any need for a known structural causal model, and that this proxy…

desk verdict A genuinely new way to shape curricula with approximate causal mismatch—but the main experiment's speedup can't yet be pinned on causality, and the abstract overstates one result. read the letter →

arxiv 2507.02910 v1 pith:7L2VSLCK submitted 2025-06-24 cs.LG cs.AIstat.ML

Geonwoo Cho , Jaegyun Im , Doyoon Kim , Sundong Kim This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords curriculumreinforcementlearningstructuralcausalmodelsensembledisagreementoptimaltransportmisalignmentscoretasknoveltysampleefficiencydeep
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CP-DRL (Causal-Paced Deep Reinforcement Learning) proposes that the causal difference between tasks can be read off ordinary interaction data: train small ensembles to reconstruct state and action and to predict transitions and rewards, then use the spread among ensemble members as a causal-misalignment score. Adding that novelty signal to the agent's recent reward gain and feeding the sum into an optimal-transport curriculum update yields curricula that converge faster and with lower variance than existing methods on the Point Mass and Bipedal Walker benchmarks, without access to ground-truth causal models. The paper's own Sparse Goal Reaching experiment shows the same signal turns into noise when tasks differ only by goal location, so the benefit is conditional on tasks actually differing in structure.

What carries the argument

The load-bearing object is the modular ensemble: state and action autoencoders trained with a $\beta$-VAE loss, a transition predictor, and a reward predictor, each replicated $K=10$ times. Disagreement is the standard deviation of predictions across ensemble members, $\text{Disagreement}_i = \mathrm{std}\{\hat{y}^{(k)}_i(\cdot)\}_{k=1}^{K}$, and a weighted sum over the four components forms the causal misalignment score $CM(c)$. The per-task cost $R_j + CM(c_j)$ is fed into CURROT's optimal-transport context update, which moves the task sampling distribution toward the target while keeping consecutive distributions close and requiring a minimal per-task return. High disagreement is read as structural unfamiliarity, so the teacher shifts sampling mass toward tasks the agent's models do not yet agree on.

What would settle it

Run the same modular ensembles on two causally identical tasks that differ only in goal position (as in Sparse Goal Reaching) and check whether transition and reward disagreement stay flat. If those disagreements rise with goal position or shape the curriculum, the signal is measuring noise, not causal difference.

Watch

Extended reading notes

Core claim

The central claim is that, in deterministic MDPs, the causal difference between two tasks can be approximated by the difference in four constituents—state, action, transition, and reward—and that this approximation can be computed from data by training an ensemble of $K$ predictors per component and taking the standard deviation of their predictions as disagreement. This yields the causal misalignment score $CM(c)=\sum_i w_i\,\text{Disagreement}_i$, which CP-DRL adds to the episodic return as a per-task cost inside CURROT's constrained optimal-transport curriculum update. On Point Mass, CP-DRL reaches $6.17\pm0.08$ at epoch 195, about 10.2% above CURROT's $5.6\pm0.34$, with low throughout-training variance; on Bipedal Walker Trivial it converges fastest with reduced variance, and on the Infeasible variant it posts the highest mean return of $130.61\pm9.32$ at 30k steps. In Sparse Goal Reaching, where causal structure is invariant, the disagreement signal is mostly noise and CP-DRL underperforms CURROT.

Load-bearing premise

The method stands or falls on the premise that, in a deterministic MDP, causal differences between tasks are fully captured by state, action, transition, and reward changes, and that the spread among quickly trained ensemble models is a faithful measure of those changes rather than of fitting noise.

Editorial extensions

If this is right

  • Curriculum design no longer requires a ground-truth causal model: disagreement among cheap modular predictors is a sufficient novelty signal to guide task selection.
  • On Point Mass, the curriculum closes to the target gate position and narrows gate width faster than CURROT, yielding about 10.2% higher final return.
  • On Bipedal Walker, CP-DRL trades a small final-return gap in the Trivial setting for markedly lower variance and faster convergence, and reaches the highest mean return mid-training in the Infeasible setting.
  • When structural variation is absent, as in Sparse Goal Reaching, the causal signal adds only noise, so a safe deployment should fall back to non-causal curriculum signals in structurally flat task families.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the disagreement proxy is valid, the same per-component ensembles can be reused as a cheap diagnostic for dynamics or representation shift in continual RL settings beyond curricula.
  • The SGR failure suggests a selection rule: run CP-DRL's causal cost only when task variation is known to alter dynamics or reward; otherwise fall back to a non-causal curriculum cost.
  • A noise-aware extension would separate aleatoric from epistemic disagreement in stochastic environments; without it, the CM score likely tracks environment noise, which the deterministic-MDP assumption currently excludes.
  • The component weights are hand-set; an adaptive or learned weighting (for example, up-weighting transition disagreement and down-weighting reward disagreement) could remove the main tuning burden and improve transfer to new environments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes CP-DRL, a curriculum reinforcement learning method that attempts to approximate differences between tasks' structural causal models from interaction data only. It trains modular ensembles of state, action, transition, and reward predictors, defines a Causal Misalignment (CM) score as a weighted sum of ensemble disagreements, and adds this score to the episodic return inside the CURROT optimal-transport curriculum objective. Experiments are reported on Point Mass, Bipedal Walker (Trivial and Infeasible), and Sparse Goal Reaching, along with a validation study in CausalWorld and a sensitivity analysis in Appendix E. The authors report faster convergence on Point Mass, reduced variance in Bipedal Walker-Trivial, and a high mid-training peak in Bipedal Walker-Infeasible, and they release full code.

Significance. If the causal interpretation were established, CP-DRL would be a useful contribution: it would replace the ground-truth SCM assumption of causally aligned curriculum learning with a data-driven proxy, potentially broadening the applicability of structure-aware curricula to realistic RL settings. The paper has clear strengths: the full implementation is released, multiple random seeds are used, the appendices transparently report a failure case (SGR), and the component-wise sensitivity analysis is a sensible diagnostic step. However, the central claim that the CM score measures causal-structure difference is not yet supported. The signal in the main benchmark reduces to transition-ensemble disagreement in a stochastic environment, and the paper provides no control separating causal misalignment from aleatoric noise, initialization variance, or generic input distribution shift. The published claims therefore outrun the evidence.

major comments (5)
  1. [§2.2, App. B.1, Table 1, Eq. (5)] The deterministic assumption in §2.2 is contradicted by the Point Mass environment description in Appendix B.1, where the Euler update includes Gaussian noise epsilon ~ N(0, 0.05^2). Since Table 1 sets the PM weights to w_state = w_action = w_reward = 0 and w_transition = 10, the CM score in Eq. (5) is exactly 10 times the standard deviation of transition-predictor ensembles on noisy transitions. Eq. (4) therefore conflates aleatoric noise and ensemble initialization variance with structural change. The PM improvement over CURROT cannot be attributed to causal-structure awareness without a matched non-causal control, for example a CURROT baseline with a generic uncertainty bonus (value disagreement or transition-prediction error) of identical architecture. Please add such an ablation and quantify how much of the PM gain remains when the causal interpretation is removed.
  2. [App. E and App. F] The sensitivity analysis in Appendix E shows that disagreement tracks block size, action scale, and reward scale, but all of these manipulations also change the marginal state and action distributions seen by the predictors, so the reported monotonic trends are also consistent with ordinary out-of-distribution input shift rather than with changes in causal mechanism. Appendix F is more direct: when only the goal location changes and the causal structure is fixed, the authors state that disagreement is "primarily driven by noise" and CP-DRL "degenerates into a noisier variant of CURROT." This is exactly the failure mode one would expect if the PM signal is generic uncertainty rather than causal misalignment. The paper needs a positive control that changes the mechanism while holding the input distribution fixed, or an analysis showing that the PM transition disagreement tracks structural change beyond what OOD input shift alone would produce.
  3. [§4.2, Abstract] The abstract claims CP-DRL "achieves the highest average performance in the Infeasible variant," but §4.2 reports that CURROT's final return is 123.58 ± 5.72 versus CP-DRL's 101.85 ± 13.87. CP-DRL's 130.61 ± 9.32 is a mid-training peak at 30k steps, not a final or overall average improvement. Please either rephrase the abstract to say "highest peak mid-training return" or provide an aggregate measure (e.g., mean return over the full training period) that supports "highest average performance." The current wording overstates the result.
  4. [Algorithm 1, line 15] The method adds CM(c_j) directly to the episodic return R_j inside the CURROT cost. The relative scale of the two terms is unspecified, and on Point Mass, where returns are small and w_transition = 10, the CM term may dominate the transport objective. No sensitivity analysis or normalization is reported for this additive combination. Please report the typical numerical scales of R_j and CM(c_j) for each environment, and include an ablation that varies the combination (e.g., multiplicative weighting or normalization) to show that the reported behavior is not driven by an accidentally chosen scale.
  5. [§4, Fig. 6 and Fig. 7] The claims of "faster convergence," "reduced variance," and "comparable final returns" rely on visual inspection of mean curves, but no significance tests or per-seed scatter are reported. This matters especially in Bipedal Walker-Infeasible, where only 3 seeds are used and the confidence intervals are wide. Please report significance tests (e.g., bootstrap or Mann-Whitney on final returns) or show individual seed curves so that the stability and variance claims can be assessed quantitatively.
minor comments (6)
  1. [Author affiliation] The affiliation contains a typo: "Gwangju Intsitute of Science and Technology" should be "Gwangju Institute of Science and Technology."
  2. [§2.2] The action space is denoted X in the SCM paragraph but A in the MDP definition in §2.1 and elsewhere; please use a consistent symbol throughout.
  3. [Figure 1] In the diagram label "CausalDifference" should be split into "Causal Difference" to match the prose.
  4. [Fig. 6 and Fig. 7 legends] The legend labels "CURROT (best method)" are confusing because CURROT is not the best final method in the Infeasible setting; relabel as "strongest baseline" or similar.
  5. [Fig. 4] The CausalWorld validation states that disagreement metrics are averaged over three random seeds, but Figure 4 shows no error bars or confidence intervals; please add them or state explicitly that only means are reported.
  6. [Algorithm 1] Please clarify whether the ensemble models are trained incrementally across curriculum iterations and whether the disagreement for a sampled context is computed before or after the ensemble is updated on that context's trajectory; this timing affects the interpretation of CM as "unfamiliarity."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CP-DRL's causal misalignment signal is an empirically evaluated proxy, not a fitted or self-referential input to the reported performance claims.

full rationale

The paper's derivation chain separates inputs from claimed outputs. The CM score (Eq. 5) is defined as a weighted sum of ensemble prediction standard deviations (Eq. 4) computed from interaction data; it is not fit to, nor defined in terms of, the final returns it is claimed to improve. The curriculum objective (Algorithm 1, line 15) combines episodic return R_j with CM(c_j) inside CURROT, and the reported PM/BW results are external benchmark comparisons against unrelated baselines (CURROT, SPRL, GoalGAN, etc.), so the empirical gains are not forced by construction. No load-bearing self-citation appears: the framework builds on external prior work (Li et al. 2024; Klink et al. 2022; Zhang et al. 2020), and none of the authors' own prior results are invoked to force a conclusion. The toy CausalWorld validation (Section 3.2 and Appendix E) tests whether modular disagreement tracks manipulated task factors; this is an empirical sensitivity check, not an equation-level circularity. Appendix F even reports a failure mode (SGR) where CM degenerates into a noisier variant of CURROT, which is inconsistent with the hypothesis that the positive PM results are trivially forced by the construction of the signal. Concerns that 'causal misalignment' is a renamed generic uncertainty signal, or that the PM setting violates the deterministic assumption underlying the SCM decomposition, are validity and ablation concerns rather than demonstrations that a prediction reduces to its input. Under the hard rules, no circular step can be exhibited with a specific equation-level reduction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a chain of assumptions: deterministic MDPs, the SCM-MDP equivalence from Li et al. (2024), ensemble disagreement as a proxy for SCM distance, and the additive combination of disagreement with return in the CURROT objective. The component weights w_i are hand-set per environment, and no normalization between return and disagreement is specified.

free parameters (2)
  • Component weights w_i for CM score = PM: w_transition=10, w_state=w_action=w_reward=0; BW: w_state=100, w_action=20, w_transition=100, w_reward=100
    Hand-chosen per environment (Appendix C). This means in Point Mass the 'causal misalignment' is purely transition disagreement.
  • Relative scale of return and CM score in CURROT cost = Implicitly 1 (R_j + CM(c_j))
    The unified objective adds episodic return and disagreement without normalization or scaling (Algorithm 1, line 15). Since these quantities have different units and magnitudes, the effective weighting is an untuned free parameter.
assumptions (4)
  • domain assumption The MDP is deterministic with no exogenous variables (U = empty).
    Section 2.2: 'we assume a deterministic setting without exogenous variables'. Needed to equate MDP with SCM and to decompose causal difference into state/action/transition/reward components. Contradicted by PM transition noise in Appendix B.1.
  • domain assumption SCM difference between tasks is fully represented by differences in state, action, transition, and reward components.
    Adopted from Li et al. (2024) without proof; used in Eq. 5 to define CM(c).
  • ad hoc to paper Ensemble disagreement is a valid proxy for structural unawareness / causal misalignment.
    Section 3.1: 'we interpret as causal misalignment'. Only toy validation with 3 seeds; reward disagreement fails to follow the expected trend.
  • ad hoc to paper Adding CM to episodic return in the CURROT cost preserves the curriculum's constraints and objectives.
    Algorithm 1 line 15 uses cost R_j + CM(c_j) without derivation, scale analysis, or discussion of how the threshold constraint in Eq. 2 is affected.
invented entities (1)
  • Causal Misalignment (CM) score
    purpose: Quantify a task's structural novelty relative to learned experience; used as additive cost in curriculum optimization.
    Defined in Eq. 5; validated only in a toy CausalWorld setup with 3 seeds and no error bars, and shown in Appendix F to degenerate to noise when tasks differ only by goal location.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Causal-Paced Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/7L2VSLCK

@misc{pith2026250702910,
  author       = {Pith},
  title        = {Pith review of: Causal-Paced Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7L2VSLCK}},
  note         = {Machine review of arXiv:2507.02910}
}
read the original abstract

Designing effective task sequences is crucial for curriculum reinforcement learning (CRL), where agents must gradually acquire skills by training on intermediate tasks. A key challenge in CRL is to identify tasks that promote exploration, yet are similar enough to support effective transfer. While recent approach suggests comparing tasks via their Structural Causal Models (SCMs), the method requires access to ground-truth causal structures, an unrealistic assumption in most RL settings. In this work, we propose Causal-Paced Deep Reinforcement Learning (CP-DRL), a curriculum learning framework aware of SCM differences between tasks based on interaction data approximation. This signal captures task novelty, which we combine with the agent's learnability, measured by reward gain, to form a unified objective. Empirically, CP-DRL outperforms existing curriculum methods on the Point Mass benchmark, achieving faster convergence and higher returns. CP-DRL demonstrates reduced variance with comparable final returns in the Bipedal Walker-Trivial setting, and achieves the highest average performance in the Infeasible variant. These results indicate that leveraging causal relationships between tasks can improve the structure-awareness and sample efficiency of curriculum reinforcement learning. We provide the full implementation of CP-DRL to facilitate the reproduction of our main results at https://github.com/Cho-Geonwoo/CP-DRL.

Figures

Figures reproduced from arXiv: 2507.02910 by the authors.

Figure 1
Figure 1. An overview of the CP-DRL. We estimate the causal difference between tasks based on the observed trajectories in each task. This structural signal is used by the teacher to construct a curriculum that gradually exposes the student to novel tasks. framework that addresses this limitation by approximating the SCM difference between tasks using only interaction data. Specifically, we estimate the differences in state, … view at source ↗
Figure 2
Figure 2. Causal graph in our RL setting. Solid arrows: environment-induced transi￾tions and reward generation, Dotted arrows: denote policy-induced dependencies. A Structural Causal Model (SCM) describes causal relationships between variables (Pearl, 2009; Pearl & Bareinboim, 2011). An SCM M is defined as a tuple (U, V, F, P), where U is a set of exogenous (unobserved) variables, V is a set of endogenous (observed) variables… view at source ↗
Figure 3
Figure 3. Visualization of tasks T1–T4 from the CausalWorld General environment. In this setup, agents receive rewards proportional to the intersection ratio between each block and the goal configuration. From T1 to T4, we progressively increase the block size, action magnitude, and reward scale, inducing increasing causal differences between tasks. To assess the effectiveness of the CM score in capturing task-level causal di… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Disagreement metrics across tasks T1–T4. Each metric is computed after training on T1 for 10 episodes and sampling 5 episodes from each task. Transition, state, and action disagreements increase monotonically, suggesting they effectively reflect causal differences. Rew…
Figure 5
Figure 5. Figure 5: Illustrations of the environments used in our experiments. (a) Point Mass. (b) Bipedal￾Walker. We evaluated our method, CP-DRL, on two benchmark environments: Point Mass (PM) and Bipedal Walker (BW). For the PM environment, we used Proximal Policy Optimization (PPO) as…
Figure 6
Figure 6. Figure 6: Performance comparison between CP-DRL and CURROT in Point Mass environ￾ment. (a) Cumulative discounted return. (b) Median distance to the target gate position. (c) Median distance to the target gate width. All curves show the mean with 95% confidence intervals. bimodal…
Figure 7
Figure 7. Figure 7: Performance comparison between CP-DRL and CURROT in BipedalWalker. (a) Trivial. (b) Infeasible. All curves show the mean with 95% confidence intervals. The BW environment is a continuous control benchmark where a bipedal agent must traverse un￾even terrain consisting o…
Figure 8
Figure 8. Figure 8: Performance comparison in Point Mass environment under different curriculum methods. (a) Cumulative discounted return over 200 epochs. (b) Median distance to the target gate position (c) Median distance to the target gate width. All curves show the mean with 95% confid…
Figure 9
Figure 9. Figure 9: Performance comparison of curriculum methods in BipedalWalker. (a) Trivial, (b) Infeasible. All curves show the mean with 95% confidence intervals. B Implementation Details B.1 Point Mass Environment The Point Mass environment presents a continuous 2-dimensional task w…
Figure 10
Figure 10. Figure 10: Component-wise disagreement under isolated changes to block size, action scale, and reward scale. any of the settings, which we attribute to the sparsity of the reward signal and the use of random trajectories during data collection. Together, these observations reinf…
Figure 11
Figure 11. Figure 11: Illustration of the Sparse Goal Reaching environment. To further investigate the limitations of CP-DRL, we conducted an additional experiment in the Sparse Goal Reaching (SGR) environment. SGR is a 2D continuous maze-like setting adapted from Florensa et al. (2018), w…
Figure 12
Figure 12. Figure 12: Performance comparison of curriculum methods in Sparse Goal Reaching. (a) Success rate (b) Median tolerance. All curves show the mean with 95% confidence intervals over 10 random seeds. Each task in this environment is defined by a context vector c = [xgoal, ygoal, ϵ]…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 31 canonical work pages

  1. [1]

    a uble, Anirudh Goyal, Alexander Neitz, Manuel W \

    Ossama Ahmed, Frederik Tr \"a uble, Anirudh Goyal, Alexander Neitz, Manuel W \"u thrich, Yoshua Bengio, Bernhard Sch \"o lkopf, and Stefan Bauer. Causalworld: A robotic manipulation benchmark for causal structure and transfer learning. In International Conference on Learning Representations, 2021

    work page 2021

  2. [2]

    Woulda, coulda, shoulda: Counterfactually-guided policy search

    Lars Buesing, Theophane Weber, Yori Zwols, Nicolas Heess, Sebastien Racaniere, Arthur Guez, and Jean-Baptiste Lespiau. Woulda, coulda, shoulda: Counterfactually-guided policy search. In International Conference on Learning Representations, 2018

    work page 2018

  3. [3]

    Minimalistic gridworld environment for gymnasium

    Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium. https://github.com/Farama-Foundation/Minigrid, 2018. Accessed: 2025-06-01

    work page 2018

  4. [4]

    Diversify & Conquer : Outcome-directed curriculum rl via out-of-distribution disagreement

    Daesol Cho, Seungjae Lee, and H Jin Kim. Diversify & Conquer : Outcome-directed curriculum rl via out-of-distribution disagreement. Advances in Neural Information Processing Systems, 36: 0 53593--53623, 2023 a

    work page 2023

  5. [5]

    Outcome-directed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation

    Daesol Cho, Seungjae Lee, and H Jin Kim. Outcome-directed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation. arXiv preprint arXiv:2301.11741, 2023 b

    arXiv 2023

  6. [6]

    Reverse curriculum generation for reinforcement learning

    Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on Robot Learning, pp.\ 482--495. PMLR, 2017

    work page 2017

  7. [7]

    Automatic goal generation for reinforcement learning agents

    Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In Proceedings of the International Conference on Machine Learning, pp.\ 1515--1528. PMLR, 2018

    work page 2018

  8. [8]

    Automated curriculum learning for neural networks

    Alex Graves, Marc G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In Proceedings of the International Conference on Machine Learning, pp.\ 1311--1320. PMLR, 2017

    work page 2017

Show all 39 references
  1. [9]

    Contextual markov decision processes

    Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.02259, 2015

    2015 arXiv

  2. [10]

    beta-vae: Learning basic visual concepts with a constrained variational framework

    Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017

    2017

  3. [11]

    Curriculum reinforcement learning using optimal transport via gradual domain adaptation

    Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum reinforcement learning using optimal transport via gradual domain adaptation. Advances in Neural Information Processing Systems, 35: 0 10656--10670, 2022

    2022

  4. [12]

    Open-endedness is essential for artificial superhuman intelligence

    Edward Hughes, Michael Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rockt \"a schel. Open-endedness is essential for artificial superhuman intelligence. arXiv preprint arXiv:2406.04268, 2024

    2024 arXiv

  5. [13]

    Prioritized level replay

    Minqi Jiang, Edward Grefenstette, and Tim Rockt \"a schel. Prioritized level replay. In Proceedings of the International Conference on Machine Learning, pp.\ 4940--4950. PMLR, 2021

    2021

  6. [14]

    Self-paced contextual reinforcement learning

    Pascal Klink, Hany Abdulsamad, Boris Belousov, and Jan Peters. Self-paced contextual reinforcement learning. In Conference on Robot Learning, pp.\ 513--529. PMLR, 2020 a

    2020

  7. [15]

    Self-paced deep reinforcement learning

    Pascal Klink, Carlo D'Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 9216--9227, 2020 b

    2020

  8. [16]

    A probabilistic interpretation of self-paced learning with applications to reinforcement learning

    Pascal Klink, Hany Abdulsamad, Boris Belousov, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. A probabilistic interpretation of self-paced learning with applications to reinforcement learning. Journal of Machine Learning Research, 22 0 (182): 0 1--52, 2021 a

    2021

  9. [17]

    Boosted curriculum reinforcement learning

    Pascal Klink, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Boosted curriculum reinforcement learning. In International Conference on Learning Representations, 2021 b

    2021

  10. [18]

    Curriculum reinforcement learning via constrained optimal transport

    Pascal Klink, Haoyi Yang, Carlo D’Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. In Proceedings of the International Conference on Machine Learning, pp.\ 11341--11358. PMLR, 2022

    2022

  11. [19]

    On the benefit of optimal transport for curriculum reinforcement learning

    Pascal Klink, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. On the benefit of optimal transport for curriculum reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  12. [20]

    Causally aligned curriculum learning

    Mingxuan Li, Junzhe Zhang, and Elias Bareinboim. Causally aligned curriculum learning. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=hp4yOjhwTs

    2024

  13. [21]

    Curriculum learning for reinforcement learning domains: A framework and survey

    Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21 0 (181): 0 1--50, 2020

    2020

  14. [22]

    Evolving curricula with regret-based environment design

    Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rockt \"a schel. Evolving curricula with regret-based environment design. In Proceedings of the International Conference on Machine Learning, pp.\ 17473--17498. PMLR, 2022

    2022

  15. [23]

    Causality: Models, Reasoning, and Inference

    Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2 edition, 2009. doi:10.1017/CBO9780511803161

    2009 doi

  16. [24]

    Transportability of causal and statistical relations: A formal approach

    Judea Pearl and Elias Bareinboim. Transportability of causal and statistical relations: A formal approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp.\ 247--254, 2011

    2011

  17. [25]

    Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments

    R \'e my Portelas, C \'e dric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pp.\ 835--853. PMLR, 2020

    2020

  18. [26]

    Teachmyagent: a benchmark for automatic curriculum learning in deep rl

    Cl \'e ment Romac, R \'e my Portelas, Katja Hofmann, and Pierre-Yves Oudeyer. Teachmyagent: a benchmark for automatic curriculum learning in deep rl. In Proceedings of the International Conference on Machine Learning, pp.\ 9052--9063. PMLR, 2021

    2021

  19. [27]

    Diffusion-based curriculum reinforcement learning

    Erdi Sayar, Giovanni Iacca, Ozgur S Oguz, and Alois Knoll. Diffusion-based curriculum reinforcement learning. Advances in Neural Information Processing Systems, 37: 0 97587--97617, 2024

    2024

  20. [28]

    Schrader

    Max-Philipp B. Schrader. Gym-sokoban. https://github.com/mpSchrader/gym-sokoban, 2018. Accessed: 2025-06-04

    2018

  21. [29]

    Causal influence detection for improving efficiency in reinforcement learning

    Maximilian Seitzer, Bernhard Sch \"o lkopf, and Georg Martius. Causal influence detection for improving efficiency in reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 22905--22918, 2021

    2021

  22. [30]

    Causal curiosity: Rl agents discovering self-supervised experiments for causal representation learning

    Sumedh A Sontakke, Arash Mehrjou, Laurent Itti, and Bernhard Sch \"o lkopf. Causal curiosity: Rl agents discovering self-supervised experiments for causal representation learning. In Proceedings of the International Conference on Machine Learning, pp.\ 9848--9858. PMLR, 2021

    2021

  23. [31]

    Model-based transfer reinforcement learning based on graphical model representations

    Yuewen Sun, Kun Zhang, and Changyin Sun. Model-based transfer reinforcement learning based on graphical model representations. IEEE Transactions on Neural Networks and Learning Systems, 2021

    2021

  24. [32]

    Proximal curriculum for reinforcement learning agents

    Georgios Tzannetos, B \'a rbara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. Proximal curriculum for reinforcement learning agents. Transactions on Machine Learning Research, 2023

    2023

  25. [33]

    Towards skilled population curriculum for multi-agent reinforcement learning

    Rundong Wang, Longtao Zheng, Wei Qiu, Bowei He, Bo An, Zinovi Rabinovich, Yujing Hu, Yingfeng Chen, Tangjie Lv, and Changjie Fan. Towards skilled population curriculum for multi-agent reinforcement learning. arXiv preprint arXiv:2302.03429, 2023

    2023 arXiv

  26. [34]

    Robust deep reinforcement learning through bootstrapped opportunistic curriculum

    Junlin Wu and Yevgeniy Vorobeychik. Robust deep reinforcement learning through bootstrapped opportunistic curriculum. In Proceedings of the International Conference on Machine Learning, pp.\ 24177--24211. PMLR, 2022

    2022

  27. [35]

    A survey on causal discovery: Theory and practice

    Alessio Zanga, Elif Ozkirimli, and Fabio Stella. A survey on causal discovery: Theory and practice. International Journal of Approximate Reasoning, 151: 0 101--129, 2022

    2022

  28. [36]

    A survey on causal reinforcement learning

    Yan Zeng, Ruichu Cai, Fuchun Sun, Libo Huang, and Zhifeng Hao. A survey on causal reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2024

    2024

  29. [37]

    Automatic curriculum learning through value disagreement

    Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. Advances in Neural Information Processing Systems, 33: 0 7648--7659, 2020

    2020

  30. [38]

    Invariant action effect model for reinforcement learning

    Zheng-Mao Zhu, Shengyi Jiang, Yu-Ren Liu, Yang Yu, and Kun Zhang. Invariant action effect model for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 9260--9268, 2022

    2022

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.