{"id":"0c26d5eb-eed5-4f1c-9ee3-d401b9ce797e","arxiv_id":"2507.02910","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CP-DRL adds an ensemble-disagreement-based causal misalignment score to the CURROT curriculum objective and reports faster convergence and higher returns on Point Mass and Bipedal Walker benchmarks.","lead":"This paper proposes a curriculum learning method for reinforcement learning that uses disagreement among an ensemble of world models to estimate how structurally novel a task is, and combines that signal with reward to select training tasks. The method is tested on two continuous control benchmarks and is claimed to reach target performance faster and with lower variance than existing curricula.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Point Mass CM reduces to transition-ensemble disagreement in a stochastic environment; without a non-causal control ablation, the PM gain over CURROT cannot be attributed to causal structure.","rationale":"The reader's conditional verdict is appropriate, and my stress-test converges on the same load-bearing assumption: the CM score in Eq. 5 is only meaningful if ensemble disagreement tracks task-level causal structure. The paper explicitly assumes determinism in Section 2.2 but evaluates on stochastic Point Mass (Appendix B.1), and Appendix C zeros out all components except transition disagreement in PM. The SGR appendix is an existence proof that the signal can be pure noise, and the paper's own explanation is that disagreement is driven by noise when causal structure is unchanged. No baseline in the paper includes a matched-magnitude non-causal uncertainty bonus, so the PM gain over CURROT cannot yet be attributed to causal structure. I would therefore keep the verdict at CONDITIONAL, with the additional requirement of a non-causal control ablation. If that ablation shows no difference, the abstract's causal claim would need to be weakened. The Infeasible summary inconsistency is real but secondary; the main concern is causal attribution. Credit is due for releasing the implementation and for reporting the negative SGR result, which makes the limitation visible and testable.","tokens_in":14452,"tokens_out":7561,"duration_ms":86407,"concrete_test":"Re-run the Point Mass experiment (Section 4.1) with the CM term in Algorithm 1 line 15 replaced by a matched-magnitude non-causal signal: the same transition-ensemble disagreement, but computed after randomly permuting task labels across the collected trajectories so the score carries no information about which task is being evaluated. Keep all other components, seeds, and CM magnitudes identical. If CP-DRL retains its roughly 10% margin over CURROT under permuted labels, the improvement is not caused by causal structure; if the margin disappears, the causal interpretation survives this control.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the CM score (Eq. 5) to measure causal-structure difference. Section 2.2 assumes deterministic MDPs (\"we assume a deterministic setting without exogenous variables\"), but the main Point Mass benchmark adds Gaussian transition noise (Appendix B.1: eps ~ N(0,0.05^2) in the Euler update). Under Appendix C Table 1, the PM weights are w_state=w_action=w_reward=0 and w_transition=10, so CM(c) is exactly 10 times the standard deviation of an ensemble of transition predictors fitted to noisy transitions (Eq. 4). Ensemble disagreement on finite noisy data conflates structural change with aleatoric noise and initialization variance, and the paper provides no experiment separating these. The paper's own Appendix F shows that when task structure is unchanged (SGR), disagreement is \"primarily driven by noise\" and CP-DRL \"degenerates into a noisier variant of CURROT\"; that is exactly the failure mode expected if the PM signal is generic uncertainty rather than causal misalignment. Because no baseline replaces the CM term with a matched non-causal uncertainty bonus, the PM speedup cannot yet be attributed to causal structure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CP-DRL, a curriculum reinforcement learning method that attempts to approximate differences between tasks' structural causal models from interaction data only. It trains modular ensembles of state, action, transition, and reward predictors, defines a Causal Misalignment (CM) score as a weighted sum of ensemble disagreements, and adds this score to the episodic return inside the CURROT optimal-transport curriculum objective. Experiments are reported on Point Mass, Bipedal Walker (Trivial and Infeasible), and Sparse Goal Reaching, along with a validation study in CausalWorld and a sensitivity analysis in Appendix E. The authors report faster convergence on Point Mass, reduced variance in Bipedal Walker-Trivial, and a high mid-training peak in Bipedal Walker-Infeasible, and they release full code.","tokens_in":14883,"tokens_out":5433,"duration_ms":64015,"significance":"If the causal interpretation were established, CP-DRL would be a useful contribution: it would replace the ground-truth SCM assumption of causally aligned curriculum learning with a data-driven proxy, potentially broadening the applicability of structure-aware curricula to realistic RL settings. The paper has clear strengths: the full implementation is released, multiple random seeds are used, the appendices transparently report a failure case (SGR), and the component-wise sensitivity analysis is a sensible diagnostic step. However, the central claim that the CM score measures causal-structure difference is not yet supported. The signal in the main benchmark reduces to transition-ensemble disagreement in a stochastic environment, and the paper provides no control separating causal misalignment from aleatoric noise, initialization variance, or generic input distribution shift. The published claims therefore outrun the evidence.","major_comments":[{"comment":"The deterministic assumption in §2.2 is contradicted by the Point Mass environment description in Appendix B.1, where the Euler update includes Gaussian noise epsilon ~ N(0, 0.05^2). Since Table 1 sets the PM weights to w_state = w_action = w_reward = 0 and w_transition = 10, the CM score in Eq. (5) is exactly 10 times the standard deviation of transition-predictor ensembles on noisy transitions. Eq. (4) therefore conflates aleatoric noise and ensemble initialization variance with structural change. The PM improvement over CURROT cannot be attributed to causal-structure awareness without a matched non-causal control, for example a CURROT baseline with a generic uncertainty bonus (value disagreement or transition-prediction error) of identical architecture. Please add such an ablation and quantify how much of the PM gain remains when the causal interpretation is removed.","section":"§2.2, App. B.1, Table 1, Eq. (5)"},{"comment":"The sensitivity analysis in Appendix E shows that disagreement tracks block size, action scale, and reward scale, but all of these manipulations also change the marginal state and action distributions seen by the predictors, so the reported monotonic trends are also consistent with ordinary out-of-distribution input shift rather than with changes in causal mechanism. Appendix F is more direct: when only the goal location changes and the causal structure is fixed, the authors state that disagreement is \"primarily driven by noise\" and CP-DRL \"degenerates into a noisier variant of CURROT.\" This is exactly the failure mode one would expect if the PM signal is generic uncertainty rather than causal misalignment. The paper needs a positive control that changes the mechanism while holding the input distribution fixed, or an analysis showing that the PM transition disagreement tracks structural change beyond what OOD input shift alone would produce.","section":"App. E and App. F"},{"comment":"The abstract claims CP-DRL \"achieves the highest average performance in the Infeasible variant,\" but §4.2 reports that CURROT's final return is 123.58 ± 5.72 versus CP-DRL's 101.85 ± 13.87. CP-DRL's 130.61 ± 9.32 is a mid-training peak at 30k steps, not a final or overall average improvement. Please either rephrase the abstract to say \"highest peak mid-training return\" or provide an aggregate measure (e.g., mean return over the full training period) that supports \"highest average performance.\" The current wording overstates the result.","section":"§4.2, Abstract"},{"comment":"The method adds CM(c_j) directly to the episodic return R_j inside the CURROT cost. The relative scale of the two terms is unspecified, and on Point Mass, where returns are small and w_transition = 10, the CM term may dominate the transport objective. No sensitivity analysis or normalization is reported for this additive combination. Please report the typical numerical scales of R_j and CM(c_j) for each environment, and include an ablation that varies the combination (e.g., multiplicative weighting or normalization) to show that the reported behavior is not driven by an accidentally chosen scale.","section":"Algorithm 1, line 15"},{"comment":"The claims of \"faster convergence,\" \"reduced variance,\" and \"comparable final returns\" rely on visual inspection of mean curves, but no significance tests or per-seed scatter are reported. This matters especially in Bipedal Walker-Infeasible, where only 3 seeds are used and the confidence intervals are wide. Please report significance tests (e.g., bootstrap or Mann-Whitney on final returns) or show individual seed curves so that the stability and variance claims can be assessed quantitatively.","section":"§4, Fig. 6 and Fig. 7"}],"minor_comments":[{"comment":"The affiliation contains a typo: \"Gwangju Intsitute of Science and Technology\" should be \"Gwangju Institute of Science and Technology.\"","section":"Author affiliation"},{"comment":"The action space is denoted X in the SCM paragraph but A in the MDP definition in §2.1 and elsewhere; please use a consistent symbol throughout.","section":"§2.2"},{"comment":"In the diagram label \"CausalDifference\" should be split into \"Causal Difference\" to match the prose.","section":"Figure 1"},{"comment":"The legend labels \"CURROT (best method)\" are confusing because CURROT is not the best final method in the Infeasible setting; relabel as \"strongest baseline\" or similar.","section":"Fig. 6 and Fig. 7 legends"},{"comment":"The CausalWorld validation states that disagreement metrics are averaged over three random seeds, but Figure 4 shows no error bars or confidence intervals; please add them or state explicitly that only means are reported.","section":"Fig. 4"},{"comment":"Please clarify whether the ensemble models are trained incrementally across curriculum iterations and whether the disagreement for a sampled context is computed before or after the ensemble is updated on that context's trajectory; this timing affects the interpretation of CM as \"unfamiliarity.\"","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a curriculum RL or general ML venue, and the authors have been unusually transparent in reporting a failure case in Appendix F. My main concern for the editor is that the causal vocabulary substantially outruns the evidence: the main PM result is consistent with a generic ensemble-uncertainty bonus in a stochastic environment, and the SGR appendix shows exactly the degeneracy one would expect if the signal were noise-driven. This is fixable, but it requires a non-causal control and a meaningful revision of the abstract and conclusions. I saw no indication of misconduct; the issue is the strength of the empirical support for the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new bit is the packaging: instead of requiring a known SCM like Li et al., CP-DRL uses ensemble disagreement over state, action, transition, and reward predictors as a proxy for causal misalignment, and plugs that into CURROT's optimal transport objective. That is a reasonable and testable idea, and the paper does some things well. The CausalWorld toy experiment is a nice sanity check, showing monotonic disagreement with induced changes in block size and action scale. The code is public, the hyperparameters are spelled out, and Appendix F honestly documents a failure mode where the signal degenerates into noise. That kind of transparency earns credit.\n\nThe soft spots are real, and one is load-bearing. The method is motivated by a deterministic SCM decomposition (Section 2.2), but the Point Mass benchmark injects Gaussian transition noise (Appendix B.1). In that environment the weights are set so CM is exactly 10 times transition disagreement, which means the signal is an ensemble of noisy transition predictors. That conflates missing causal structure with aleatoric noise and random initialization. The paper does not run a matched control—replace the CM term with a non-causal uncertainty bonus like count-based exploration or RND—so the PM speedup over CURROT cannot yet be attributed to causal structure. Appendix F makes this worse: when the causal structure is unchanged, CP-DRL becomes a noisier CURROT. That is precisely the pattern you would see if the signal were generic uncertainty.\n\nThere is also a clear overclaim. The abstract and intro say CP-DRL achieves the highest average performance in the Infeasible variant, and the full paper clarifies that this refers to mid-training. The final numbers are CURROT 123.58 vs CP-DRL 101.85; CP-DRL's early peak is not the same as winning. The text in Section 4.2 is honest about this, but the abstract is not. Minor issues: no significance tests, component weights are chosen per environment, and the scaling between return and CM score inside the CURROT cost is not described, which hurts reproducibility outside the two chosen benchmarks.\n\nWho should read this? People working on curriculum RL, especially anyone interested in combining structure-aware signals with optimal transport curricula. The paper is worth a serious referee—the idea is clearly stated, the code exists, and the empirical questions are addressable. I would send it to review, but with the expectation of major revision: add a non-causal control, fix the abstract, and tighten the causal language.","headline":"A genuinely new way to shape curricula with approximate causal mismatch—but the main experiment's speedup can't yet be pinned on causality, and the abstract overstates one result.","tokens_in":787,"tokens_out":748,"would_cite":false,"duration_ms":23861,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper tries to establish that task selection in curriculum reinforcement learning can be driven by an interaction-data proxy for causal task difference, avoiding any need for a known structural causal model, and that this proxy…","keywords":["curriculum reinforcement learning","structural causal models","ensemble disagreement","optimal transport curriculum","causal misalignment score","task novelty","sample efficiency","deep reinforcement learning"],"falsifier":"Run the same modular ensembles on two causally identical tasks that differ only in goal position (as in Sparse Goal Reaching) and check whether transition and reward disagreement stay flat. If those disagreements rise with goal position or shape the curriculum, the signal is measuring noise, not causal difference.","tokens_in":14289,"feed_emoji":"🤖","tokens_out":10962,"duration_ms":106057,"temperature":0.7,"pith_summary":"CP-DRL (Causal-Paced Deep Reinforcement Learning) proposes that the causal difference between tasks can be read off ordinary interaction data: train small ensembles to reconstruct state and action and to predict transitions and rewards, then use the spread among ensemble members as a causal-misalignment score. Adding that novelty signal to the agent's recent reward gain and feeding the sum into an optimal-transport curriculum update yields curricula that converge faster and with lower variance than existing methods on the Point Mass and Bipedal Walker benchmarks, without access to ground-truth causal models. The paper's own Sparse Goal Reaching experiment shows the same signal turns into noise when tasks differ only by goal location, so the benefit is conditional on tasks actually differing in structure.","feed_headline":"Model disagreement builds faster RL curricula","feed_subtitle":"Ensemble disagreement finds causally unfamiliar tasks, beating baselines without any true causal graph.","key_machinery":"The load-bearing object is the modular ensemble: state and action autoencoders trained with a $\\beta$-VAE loss, a transition predictor, and a reward predictor, each replicated $K=10$ times. Disagreement is the standard deviation of predictions across ensemble members, $\\text{Disagreement}_i = \\mathrm{std}\\{\\hat{y}^{(k)}_i(\\cdot)\\}_{k=1}^{K}$, and a weighted sum over the four components forms the causal misalignment score $CM(c)$. The per-task cost $R_j + CM(c_j)$ is fed into CURROT's optimal-transport context update, which moves the task sampling distribution toward the target while keeping consecutive distributions close and requiring a minimal per-task return. High disagreement is read as structural unfamiliarity, so the teacher shifts sampling mass toward tasks the agent's models do not yet agree on.","core_discovery":"The central claim is that, in deterministic MDPs, the causal difference between two tasks can be approximated by the difference in four constituents—state, action, transition, and reward—and that this approximation can be computed from data by training an ensemble of $K$ predictors per component and taking the standard deviation of their predictions as disagreement. This yields the causal misalignment score $CM(c)=\\sum_i w_i\\,\\text{Disagreement}_i$, which CP-DRL adds to the episodic return as a per-task cost inside CURROT's constrained optimal-transport curriculum update. On Point Mass, CP-DRL reaches $6.17\\pm0.08$ at epoch 195, about 10.2% above CURROT's $5.6\\pm0.34$, with low throughout-training variance; on Bipedal Walker Trivial it converges fastest with reduced variance, and on the Infeasible variant it posts the highest mean return of $130.61\\pm9.32$ at 30k steps. In Sparse Goal Reaching, where causal structure is invariant, the disagreement signal is mostly noise and CP-DRL underperforms CURROT.","pith_inferences":["If the disagreement proxy is valid, the same per-component ensembles can be reused as a cheap diagnostic for dynamics or representation shift in continual RL settings beyond curricula.","The SGR failure suggests a selection rule: run CP-DRL's causal cost only when task variation is known to alter dynamics or reward; otherwise fall back to a non-causal curriculum cost.","A noise-aware extension would separate aleatoric from epistemic disagreement in stochastic environments; without it, the CM score likely tracks environment noise, which the deterministic-MDP assumption currently excludes.","The component weights are hand-set; an adaptive or learned weighting (for example, up-weighting transition disagreement and down-weighting reward disagreement) could remove the main tuning burden and improve transfer to new environments."],"forward_implications":["Curriculum design no longer requires a ground-truth causal model: disagreement among cheap modular predictors is a sufficient novelty signal to guide task selection.","On Point Mass, the curriculum closes to the target gate position and narrows gate width faster than CURROT, yielding about 10.2% higher final return.","On Bipedal Walker, CP-DRL trades a small final-return gap in the Trivial setting for markedly lower variance and faster convergence, and reaches the highest mean return mid-training in the Infeasible setting.","When structural variation is absent, as in Sparse Goal Reaching, the causal signal adds only noise, so a safe deployment should fall back to non-causal curriculum signals in structurally flat task families."],"supporting_citations":[{"why":"Supplies CURROT, the constrained optimal-transport framework whose context update CP-DRL modifies with the causal-misalignment cost; it is also the strongest baseline.","marker":"(Klink et al., 2022)"},{"why":"Introduces SCM-based curriculum comparison that needs ground-truth causal structure; CP-DRL is positioned as the interaction-data alternative.","marker":"(Li et al., 2024)"},{"why":"Provides the ensemble-disagreement idea for uncertainty in curriculum RL, which CP-DRL adapts to modular component predictors.","marker":"(Zhang et al., 2020)"},{"why":"CausalWorld is the controlled environment used to validate that component-wise disagreement rises with induced causal differences.","marker":"(Ahmed et al., 2021)"},{"why":"The beta-VAE loss is used for the state and action reconstruction modules.","marker":"(Higgins et al., 2017)"},{"why":"Defines structural causal models and the causal interpretation of state-action-reward dependencies.","marker":"(Pearl, 2009)"},{"why":"Formalizes transportability of causal relations, the conceptual basis for comparing tasks through SCM differences.","marker":"(Pearl & Bareinboim, 2011)"},{"why":"Defines the Point Mass environment and the self-paced curriculum framework underlying later methods and the PM benchmark.","marker":"(Klink et al., 2020a)"},{"why":"Source of the Sparse Goal Reaching setting and of goal-generation curriculum baselines used in the failure analysis.","marker":"(Florensa et al., 2018)"}],"fun_headline_variants":["Ensemble disagreement paces RL curricula without causal graphs","Causal misalignment from ensembles speeds curriculum RL","No causal graph? Ensemble disagreement still builds fast RL curricula","Data-only causal difference sharpens RL task sequencing","Causal-pacing: ensemble variance replaces ground-truth SCMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands or falls on the premise that, in a deterministic MDP, causal differences between tasks are fully captured by state, action, transition, and reward changes, and that the spread among quickly trained ensemble models is a faithful measure of those changes rather than of fitting noise.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble disagreement paces RL curricula without causal graphs","Causal misalignment from ensembles speeds curriculum RL","No causal graph? Ensemble disagreement still builds fast RL curricula","Data-only causal difference sharpens RL task sequencing","Causal-pacing: ensemble variance replaces ground-truth SCMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1439,"prompt_tokens":996,"completion_tokens":443,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":365}},"tokens_in":612,"tokens_out":443,"duration_ms":5150,"temperature":1.0,"reasoning_tokens":365,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:22:23.236308+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same modular ensembles on two causally identical tasks that differ only in goal position (as in Sparse Goal Reaching) and check whether transition and reward disagreement stay flat. If those disagreements rise with goal position or shape the curriculum, the signal is measuring noise, not causal difference.","supporting_citations":[{"cited_title":"Curriculum reinforcement learning via constrained optimal transport","cited_arxiv_id":null,"evidence_quote":"Supplies CURROT, the constrained optimal-transport framework whose context update CP-DRL modifies with the causal-misalignment cost; it is also the strongest baseline."},{"cited_title":"a uble, Anirudh Goyal, Alexander Neitz, Manuel W \\","cited_arxiv_id":null,"evidence_quote":"CausalWorld is the controlled environment used to validate that component-wise disagreement rises with induced causal differences."},{"cited_title":"Transportability of causal and statistical relations: A formal approach","cited_arxiv_id":null,"evidence_quote":"Formalizes transportability of causal relations, the conceptual basis for comparing tasks through SCM differences."},{"cited_title":"Automatic goal generation for reinforcement learning agents","cited_arxiv_id":null,"evidence_quote":"Source of the Sparse Goal Reaching setting and of goal-generation curriculum baselines used in the failure analysis."}],"review_version":2}