Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges

Miguel Arana-Catania; Weisi Guo

REVIEW 3 major objections 5 minor 94 references

Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Causal Curiosity accurately estimates a single causal factor in a simulated robotic-manipulation task and can refine that estimate three orders of magnitude, but the paper claims that when two factors vary simultaneously the method can…

desk verdict A genuinely useful empirical stress-test of Causal Curiosity whose central 'impossibility' claim overreaches: the evidence shows two untuned, non-adaptive planners failing, not that simultaneous multi-factor estimation is impossible. read the letter →

arxiv 2505.08453 v1 pith:VY7MTZ7M submitted 2025-05-13 cs.RO cs.LG

Miguel Arana-Catania , Weisi Guo This is my paper · ORCID

classification cs.ROcs.LG

keywords reinforcementlearningcausalanalysisparameterestimationsensitivitydynamicalsystemscuriosityroboticmanipulationconfoundingfactors

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper measures how far the Causal Curiosity reinforcement-learning approach can go as a practical tool for estimating unknown causal parameters of a dynamical system, using a simulated robot manipulator pushing an object. The paper claims that in ideal single-factor experiments the method works well: it classifies all five tested factors (mass, size, lateral friction, spinning friction, gravity) perfectly and can refine the estimate to about one-thousandth of the original range by repeated bipartition. It also finds a hard limit: when a second factor varies at the same time, identification of the target factor often fails outright, with spinning friction the hardest and size the easiest to recover. A proposed PPO-based planner plus an F1 classification term in the reward gives modest robustness gains over the original cross-entropy planner.

What carries the argument

The load-bearing device is the cluster-separation reward of Eq. (6), $R = C(q_M(o^i_{0:T}), z^i) + k \cdot S(o^i_{0:T})$, where $C$ is an F1 classification score comparing the learned clustering of object trajectories with the true parameter-range labels, $S$ is a Silhouette score measuring how well the two trajectory clusters separate (using soft dynamic time warping as the distance), and $k = 0.1$ weights separation below correctness. This reward is what the planners optimize and what the paper uses to declare success or failure: a parameter is judged estimable exactly when some action sequence makes the trajectories of the two parameter ranges form distinguishable groups. The paper compares two search mechanisms over that reward, the original open-loop cross-entropy model-predictive planner and a closed-loop policy-gradient (PPO) planner, and adds recursive bipartition of the parameter range to push resolution to very fine granularity.

What would settle it

Run the hardest multi-factor cell (spinning friction as target with size as the secondary factor) under an exhaustive grid over the six-step control sequence, or with a closed-loop policy that re-aims the robot based on live object motion: if any search finds F1 = 1.0 where the paper reports 0.55, the claimed impossibility is an artifact of the planners' search rather than a property of the physical system.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a map of where Causal Curiosity succeeds and where it breaks. The method's reward—correct classification of each trajectory plus cluster separation—is sufficient to drive accurate parameter estimation when only one factor is varied and the two ranges are well separated; under these conditions the paper reports F1 scores of 1.00 for all five factors and demonstrates refinement to ranges about 0.14% of the initial width. The paper's headline limitation is the claim that it is impossible to solve some of the newly studied multi-factor scenarios: when two uncorrelated factors vary simultaneously, the F1 score for the target factor can drop to as low as 0.55 (spinning friction against size as the secondary factor), and success depends strongly on which factor is the target and which is the confounder. Causally related and confounding factors, modelled with additive noise models, were mostly identified correctly. The paper reads these results as showing that Causal Curiosity is a generalisable method for single-factor estimation, but that robustness to simultaneous variation is system- and factor-dependent.

Load-bearing premise

The paper equates successful parameter estimation with finding a single action sequence whose object trajectories split into two well-separated clusters matching the two parameter ranges, as judged by the F1 and Silhouette scores; if a different way of choosing actions or observing trajectories could separate groups that this scheme cannot, the reported limits are limits of the scheme, not of what is knowable about the parameters.

Editorial extensions

If this is right

In controlled single-factor experiments, all five causal factors are classified with F1 = 1.00 and comparable clustering scores, so the method is factor-agnostic in easy conditions.
Repeated bipartition estimates a parameter to roughly three orders of magnitude finer than the starting range while clustering scores generally improve, implying the method can serve as a high-resolution estimator rather than a coarse classifier.
Shrinking the gap between the two parameter ranges to about 4% of their size lowers clustering scores by about a quarter on average and can cause outright misclassification, so range separation, not just range width, drives reliability.
Simultaneous uncorrelated variation of a second factor breaks identification for some factor pairs: the target factor's F1 can fall to 0.55, and factor identity matters more than the choice of planner.
Causally related and confounding factor scenarios (additive noise models) are mostly identified correctly, suggesting the method can handle structured dependence when the causal graph is known.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's own claims, the 'impossibility' result is a statement about the tested planners and observation space, not a formal identifiability proof—a closed-loop policy that conditions actions on live observations, or an observation space enriched with contact forces, might succeed where these planners fail.
The reward's geometric structure suggests a testable prediction: difficulty with two varying factors should track the overlap of the induced trajectory distributions; computing the Bayes-optimal classification error on trajectories before running the planners would settle whether the planners or the physics are the bottleneck.
The clustering score could plausibly be reused as an uncertainty estimate for adaptive partitioning, turning the agnostic bipartition into a Bayesian-style exploration strategy that splits ranges where uncertainty is highest.
The reported factor hierarchy (size easy, spinning friction hard) may serve as an evaluation target for causal representation learning: a model that captures true causal structure should be able to predict this ordering from the system's dynamics alone.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper you asked about is a useful empirical boundary study of Causal Curiosity, and the authors deserve credit for putting the method under pressure. What's genuinely new: they add an F1 classification term to the reward, use PPO as the search planner rather than only CEM, and run scenarios the original proposal didn't consider — two factors varying at once, and factors with causal relationships including a confounder. The high-precision estimation result in Section 4.2 is the strongest positive finding: they push parameter resolution three orders of magnitude finer, and PPO often beats CEM on clustering score. The Section 4.5 causal/confounding experiments are a reasonable first step, and the authors are properly cautious about extrapolating from them.

The soft spots are real and not equally serious. The main one is the 'impossibility' claim in the Introduction and Discussion. Section 4.4 shows two planners failing, not that simultaneous multi-factor estimation is impossible. The authors explicitly say hyperparameters were deliberately not tuned, and both planners are open-loop or effectively non-adaptive: the CEM planner samples a fixed action sequence, and the PPO wrapper forces all environments to execute the same actions with reward only at the last timestep. So the negative result is about the tested planners under one clustering proxy, not about identifiability of the parameters. The Discussion's 'hard limit' wording should be toned down.

Two evaluation-protocol concerns are also legitimate. Section 3.4 describes using new objects for F1 evaluation, while Section 3.6 says the same 20 trajectories are used to define the clustering function and score it. The paper should state plainly whether reported F1 is training or held-out. Most tables lack error bars or per-replication variance, which matters because several key scores (e.g., 0.85, 0.65) are near chance for 20 environments. And while the authors point to the original code, they don't ship their own modified code or data; for an empirical paper, that weakens reproducibility.

None of this is fatal. The central empirical pattern — single-factor estimation works and can be very precise, while simultaneous factor variation degrades performance — is credible. The problem is the gap between that pattern and the word 'impossibility.' Researchers applying or extending Causal Curiosity in robotics or causal RL will get a practical map of where the method breaks. It deserves serious peer review with major revision: reframe the strong claim, clarify the evaluation protocol, and ideally release the code. I would cite it as an empirical stress-test.

Referee Report

3 major / 5 minor

Summary. The paper presents an empirical evaluation of Causal Curiosity, a reinforcement-learning method for estimating causal factors (mass, size, lateral friction, spinning friction, gravity) in the CausalWorld TriFinger robotic manipulation simulator. The authors study five research questions: accuracy across different factors, granularity via repeated bipartition of factor ranges, the effect of reducing the gap between parameter ranges, simultaneous variation of multiple factors, and causally related or confounding factors. They propose augmenting the original Silhouette-based reward with an F1 classification score and compare the original CEM open-loop planner with a PPO-based planner that they adapt to share one action sequence across all environments. The reported results show perfect F1 classification in single-factor experiments, successful estimation at fine granularity, degraded clustering scores when gaps shrink, frequent failures when two factors vary simultaneously, and mostly successful estimation in the two confounding scenarios. The paper concludes that single-factor estimation is robust and highly refinable, that simultaneous variation of two causal factors represents a hard limit, and that the PPO planner offers modest robustness gains over CEM.

Significance. If the central conclusions are accepted, the paper would be a useful benchmark for Causal Curiosity, since it is, to my knowledge, the first systematic accuracy and failure-mode analysis of the method across multiple causal factors, granularities, and confounding structures. The authors are commendably explicit about their comparative rather than best-case experimental design, and the introduction of an F1 classification term is a sensible extension for detecting situations where clustering separates trajectories but not the true factor groups. The paper also provides concrete comparative evidence that factor identifiability is factor-dependent. However, the strongest claimed contribution, the 'impossibility' of multi-factor estimation, is not supported by the experimental design as presented, because the negative results come from two non-adaptive, deliberately untuned planners evaluated under a clustering proxy. The proper contribution of the paper is a careful empirical stress test of two specific planners under that proxy, and the manuscript should be revised to state that scope precisely.

major comments (3)

[§4.4, Introduction, §3.5, §3.6] The Introduction's claim that the paper shows 'the impossibility of solving some complex experimental situations ... with multiple parameters varying simultaneously' overstates what Section 4.4 can support. The negative results are obtained with exactly two planners, both of which are non-adaptive open-loop controllers: the CEM planner of §3.5, and the PPO planner that is deliberately modified so that all environments execute identical actions and receive the reward only at the last timestep (§3.5). Section 3.6 additionally states that hyperparameters were 'defined without the need for hypertuning processes' because the aim was comparative, not best-possible, results. A failure of these two untuned, non-adaptive planners under the clustering proxy of Eq. (6) is evidence about the limits of those planners, not about the identifiability of the causal factors. I recommend replacing 'impossibility' with a claim scoped to the tested planners, such as 'open-loop planners with fixed hyperparameters fail on several two-factor combinations under this clustering objective,' and, if feasible, adding a closed-loop or adaptive baseline to test whether the difficulty persists.
[§3.4, §3.6, Eq. (6)] The evaluation protocol for the F1 component of Eq. (6) is ambiguous and potentially circular. Section 3.4 describes a protocol in which the clustering function qM is learned from one set of trajectories and then used to classify new objects, but Section 3.6 states that 'the trajectories are used to define a clustering function, which allows to obtain the scores' within the same experiment, without mentioning a held-out set. If F1 is computed on the same 20 trajectories used to define qM, then the reported F1 scores measure training-set separation rather than out-of-sample classification accuracy; this would inflate the F1=1.00 results in Tables 2-4 and weaken the interpretation of the low-F1 results in Tables 5-8. The manuscript should unambiguously state whether F1 is computed on held-out trajectories, and if it is not, the analysis should be rerun with a train/test split.
[§3.6, Tables 5-8] The quantitative results are reported only as point estimates, without standard deviations or confidence intervals, even though Section 3.6 states that each experiment comprises 100 replications. This is load-bearing for the central limit claims: for example, Table 5 reports Spinning Friction F1 values of 0.65, 0.55, 0.75, and 0.65 depending on the secondary factor, and without error bars or a chance-level baseline one cannot distinguish reliable above-chance performance from an unreliable one. Reporting mean ± standard deviation over the 100 replications, together with a random-classification baseline for the 20-environment setup, would substantially strengthen the conclusions drawn from these tables.

minor comments (5)

[Throughout] The word 'planer' is used instead of 'planner' in many places (e.g., §3.5, §3.6, and Tables 2-9); a global typo correction is needed.
[§3.6] The text says the PPO planner is implemented from 'the PPO2 implementation of Stable Baselines,' while earlier in the same section Stable Baselines 3 is named; the exact library version and wrappers should be clarified for reproducibility.
[§4.5, Eq. (7)] Equation (7) renders the anticausal inequality as 'G ≠ fG(L) + N2,' which is unclear; the intended statement is presumably that G cannot be expressed as an additive function of L, and this should be written out explicitly.
[§4.4, Tables 5-8] The exclusion of Mass-Gravity combinations is justified only by saying their effects 'partially counteract each other'; a more detailed explanation is needed, since this omission changes the completeness of the factor-pair analysis.
[§4.5, Figures 9-10, Table 9] The cluster labels C1-C6 referenced in Table 9 are not marked on Figures 9 and 10; labeling the panels directly would make the table much easier to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reward-as-metric design is a standard task definition, and the paper's limit claims are empirical outcomes of specific planners rather than conclusions forced by construction.

full rationale

The paper contains no derivation chain whose output is equivalent to its input. The only potentially self-referential design choice is stated in Section 3.4: "The full reward will be used both to drive the training of our RL system and to evaluate the performance of the results." This is not load-bearing circularity because the reward (Eq. 6, F1 plus k times Silhouette) is the task definition rather than a fitted prediction: the clustering function is learned from one set of trajectories and then applied to new objects, both planners optimize the same reward, and the paper's main conclusions are the empirical successes and failures reported in Tables 2-9. The "impossibility" phrasing in Sections 1 and 5 overstates what the experiments show, since only two specific, deliberately untuned, non-adaptive planners were tested under one clustering proxy, but that is a correctness or external-validity concern, not a reduction of the conclusion to its assumptions by construction. There are no self-citations used as load-bearing evidence, no imported uniqueness theorem, and no ansatz smuggled in via citation. The central claims are empirical, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central evaluation rests on the standard causal modeling framework inherited from the literature (SCM, DAG, Markov factorization, independent mechanisms), on the faithfulness of the Bullet and CausalWorld simulator, and on the paper's own operationalization of estimation accuracy as cluster separability with a hand-picked reward weight k=0.1. No new physical entities are postulated. The causal-dependency experiments additionally assume an additive noise model with independent noise (Section 4.5).

free parameters (2)

k (reward weight) = 0.1
Used in Eq. (6) to balance the F1 classification score and the Silhouette clustering score in the reward. Chosen by the authors without reported tuning or sensitivity analysis (Section 3.4).
entropy coefficient (ent_coef) = 0
PPO hyperparameter set to zero by hand (Section 3.6). Disabling entropy regularization changes exploration and affects all PPO results reported in the tables.

assumptions (4)

domain assumption The system is acyclic and Markov-factorizable with independent mechanisms (Section 3.2)
Standard SCM assumptions from Pearl and Peters et al. invoked to justify the causal DAG in Figure 2 and the reward design.
domain assumption The Bullet physics engine inside CausalWorld faithfully simulates the TriFinger robot and cube dynamics used for all experiments (Section 3.1)
All claims about estimation accuracy and limits are claims about this simulator; transfer to real hardware is asserted, not measured.
ad hoc to paper Trajectory cluster separability (F1 plus Silhouette) is a valid measure of causal factor identifiability (Eq. 6, Section 3.4)
The paper defines estimation accuracy through this reward; the 'hard limit' conclusions in Section 4.4 inherit the assumption that failure to cluster implies failure to estimate.
domain assumption The causal relationships in Section 4.5 are accurately modeled by the discrete Additive Noise Models of Eqs. (7)-(8) with independent noise
The confounding experiments assume N1 perpendicular to G, N2 perpendicular to G, and linear additive structure; these are design choices rather than results derived from data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges." pith.science (2026). https://pith.science/paper/VY7MTZ7M

@misc{pith2026250508453,
  author       = {Pith},
  title        = {Pith review of: Parameter Estimation using Reinforcement Learning Causal Curiosity: Limits and Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VY7MTZ7M}},
  note         = {Machine review of arXiv:2505.08453}
}

read the original abstract

Causal understanding is important in many disciplines of science and engineering, where we seek to understand how different factors in the system causally affect an experiment or situation and pave a pathway towards creating effective or optimising existing models. Examples of use cases are autonomous exploration and modelling of unknown environments or assessing key variables in optimising large complex systems. In this paper, we analyse a Reinforcement Learning approach called Causal Curiosity, which aims to estimate as accurately and efficiently as possible, without directly measuring them, the value of factors that causally determine the dynamics of a system. Whilst the idea presents a pathway forward, measurement accuracy is the foundation of methodology effectiveness. Focusing on the current causal curiosity's robotic manipulator, we present for the first time a measurement accuracy analysis of the future potentials and current limitations of this technique and an analysis of its sensitivity and confounding factor disentanglement capability - crucial for causal analysis. As a result of our work, we promote proposals for an improved and efficient design of Causal Curiosity methods to be applied to real-world complex scenarios.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

94 extracted references · 40 canonical work pages

[1]

Sch¨ olkopf, B.: Causality for Machine Learning, 1st edn., pp. 765–804. Association for Computing Machin- ery, New York, NY, USA (2022). https://doi.org/10.1145/3501714.3501755

arXiv 2022
[2]

Proceedings of the IEEE 109(5), 612–634 (2021) https://doi.org/10.1109/JPROC.2021.3058954

Sch¨ olkopf, B., Locatello, F., Bauer, S., Ke, N.R., Kalchbrenner, N., Goyal, A., Bengio, Y.: Toward causal rep- resentation learning. Proceedings of the IEEE 109(5), 612–634 (2021) https://doi.org/10.1109/JPROC.2021.3058954

arXiv 2021
[3]

In: International Conference on Machine Learning, pp

Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch¨ olkopf, B., Bachem, O.: Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. In: International Conference on Machine Learning, pp. 4114–4124 (2019). PMLR

2019
[4]

In: Interna- tional Conference on Machine Learning, pp

Suter, R., Miladinovic, D., Sch¨ olkopf, B., Bauer, S.: Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In: Interna- tional Conference on Machine Learning, pp. 6056–6065 (2019). PMLR

2019
[5]

In: Proceedings of the 29th International Coference on Inter- national Conference on Machine Learning

Sch¨ olkopf, B., Janzing, D., Peters, J., Sgouritsa, E., Zhang, K., Mooij, J.: On causal and anticausal learning. In: Proceedings of the 29th International Coference on Inter- national Conference on Machine Learning. ICML’12, pp. 459–466. Omnipress, Madison, WI, USA (2012)

2012
[6]

arXiv preprint arXiv:1812.00524 (2018)

Kilbertus, N., Parascandolo, G., Sch¨ olkopf, B.: Generalization in anti-causal learning. arXiv preprint arXiv:1812.00524 (2018)

arXiv 2018
[7]

arXiv preprint arXiv:1812.10576 (2018)

Lu, C., Sch¨ olkopf, B., Hern´ andez-Lobato, J.M.: Deconfounding reinforcement learn- ing in observational settings. arXiv preprint arXiv:1812.10576 (2018)

arXiv 2018
[8]

Singla, S., Feizi, S.: Salient imagenet: How to discover spurious features in deep learn- ing? In: International Conference on Learning Representations (2022)

2022

Show all 94 references

[9]

In: Proceedings of the European Conference on Computer Vision (ECCV), pp

Beery, S., Van Horn, G., Perona, P.: Recogni- tion in terra incognita. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 456–473 (2018)

2018
[10]

In: Interna- tional Conference on Machine Learning, pp

Wang, Z., Xiao, X., Xu, Z., Zhu, Y., Stone, P.: Causal dynamics learning for task- independent state abstraction. In: Interna- tional Conference on Machine Learning, pp. 23151–23180 (2022). PMLR

2022
[11]

Advances in neural information processing systems 32 (2019)

De Haan, P., Jayaraman, D., Levine, S.: Causal confusion in imitation learning. Advances in neural information processing systems 32 (2019)

2019
[12]

arXiv preprint arXiv:2110.10819 (2021)

Ortega, P.A., Kunesch, M., Del´ etang, G., Genewein, T., Grau-Moya, J., Veness, J., Buchli, J., Degrave, J., Piot, B., Perolat, J., et al.: Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprint arXiv:2110.10819 (2021)

2021 arXiv
[13]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.- S., Wen, J.-R.: Counterfactual VQA: A cause- eﬀect look at language bias. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12700– 12710 (2021)

2021
[14]

In: Riloﬀ, E., Chiang, D., Hockenmaier, J., Tsujii, J

Alzantot, M., Sharma, Y., Elgohary, A., Ho, B.-J., Srivastava, M., Chang, K.- W.: Generating natural language adversar- ial examples. In: Riloﬀ, E., Chiang, D., Hockenmaier, J., Tsujii, J. (eds.) Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Pro...

2018 doi
[15]

In: ICML 2022: Workshop on Spurious Correla- tions, Invariance and Stability (2022) 19

Chen, Y., Zhang, Y., Bian, Y., Yang, H., KAILI, M., Xie, B., Liu, T., Han, B., Cheng, J.: Invariance principle meets out- of-distribution generalization on graphs. In: ICML 2022: Workshop on Spurious Correla- tions, Invariance and Stability (2022) 19

2022
[16]

In: Proceedings of the 44th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval

Feng, F., Huang, W., He, X., Xin, X., Wang, Q., Chua, T.-S.: Should graph con- volution trust neighbors? a simple causal inference method. In: Proceedings of the 44th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval. SIGIR ’21, pp. 1208...

2021
[17]

Cambridge University Press, Cambridge (2009)

Pearl, J.: Causality: Models, Reasoning, and Inference, 2nd edn. Cambridge University Press, Cambridge (2009)

2009
[18]

John Wiley & Sons (2016)

Glymour, M., Pearl, J., Jewell, N.P.: Causal inference in statistics: A primer. John Wiley & Sons (2016)

2016
[19]

The MIT Press (2017)

Peters, J., Janzing, D., Sch¨ olkopf, B.: Ele- ments of Causal Inference: Foundations and Learning Algorithms. The MIT Press (2017)

2017
[20]

MIT press (2000)

Spirtes, P., Glymour, C.N., Scheines, R., Heckerman, D.: Causation, prediction, and search. MIT press (2000)

2000
[21]

In: Interna- tional Conference on Machine Learning, vol

Sontakke, S.A., Mehrjou, A., Itti, L., Sch¨ olkopf, B.: Causal curiosity: RL agents discovering self-supervised experiments for causal representation learning. In: Interna- tional Conference on Machine Learning, vol. 139, pp. 9848–9858 (2021). PMLR

2021
[22]

Pre- cision Engineering 74, 347–357 (2022) https://doi.org/10.1016/j.precisioneng.2021.12.009

Thompson, J., Kasun Prasanga, D., Murakami, T.: Identiﬁcation of unknown object properties based on tactile motion sequence using 2-ﬁnger gripper robot. Pre- cision Engineering 74, 347–357 (2022) https://doi.org/10.1016/j.precisioneng.2021.12.009

2022 doi
[23]

IEEJ Journal of Industry Applications 7(6), 495–505 (2018) https://doi.org/10.1541/ieejjia.7.495

MKC, D.C., Shimono, T.: Inertia compen- sation of motion copying system for dex- terous object handling. IEEJ Journal of Industry Applications 7(6), 495–505 (2018) https://doi.org/10.1541/ieejjia.7.495

2018 doi
[24]

In: Xiao, J., Kr¨ oger, T., Khatib, O

Murali, A., Li, Y., Gandhi, D., Gupta, A.: Learning to grasp without seeing. In: Xiao, J., Kr¨ oger, T., Khatib, O. (eds.) Proceedings of the 2018 International Symposium on Exper- imental Robotics, pp. 375–386. Springer, Cham (2020)

2020
[25]

In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp

Yuan, W., Zhu, C., Owens, A., Srinivasan, M.A., Adelson, E.H.: Shape-independent hardness estimation using deep learning and a gelsight tactile sensor. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 951–958 (2017). https://doi.org/10.1109/ICRA.20...

2017
[26]

In: 2016 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pp

Yi, Z., Calandra, R., Veiga, F., Hoof, H., Hermans, T., Zhang, Y., Peters, J.: Active tactile object exploration with gaus- sian processes. In: 2016 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pp. 4925–4930 (2016). https://doi.org/10.1109/IROS...

2016
[27]

IEEE Robotics and Automa- tion Letters 2(4), 2143–2150 (2017) https://doi.org/10.1109/LRA.2017.2720853

Kaboli, M., Feng, D., Yao, K., Lanillos, P., Cheng, G.: A tactile-based frame- work for active object learning and discrimination using multimodal robotic skin. IEEE Robotics and Automa- tion Letters 2(4), 2143–2150 (2017) https://doi.org/10.1109/LRA.2017.2720853

2017
[28]

arXiv preprint arXiv:1702.02453 (2017)

Yu, W., Tan, J., Liu, C.K., Turk, G.: Prepar- ing for the unknown: Learning a univer- sal policy with online system identiﬁcation. arXiv preprint arXiv:1702.02453 (2017)

2017 arXiv
[29]

In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp

Murooka, M., Nozawa, S., Kakiuchi, Y., Okada, K., Inaba, M.: Feasibility evalua- tion of object manipulation by a humanoid robot based on recursive estimation of the object’s physical properties. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 408...

2017
[30]

Robotics and Autonomous Systems 124, 103374 (2020) https://doi.org/10.1016/j.robot.2019.103374

Mavrakis, N., Stolkin, R.: Estimation and exploitation of objects’ inertial parameters in robotic grasping and manipulation: A survey. Robotics and Autonomous Systems 124, 103374 (2020) https://doi.org/10.1016/j.robot.2019.103374

2020
[31]

arXiv preprint arXiv:1707.06347 (2017)

Schulman, J., Wolski, F., Dhariwal, P., 20 Radford, A., Klimov, O.: Proximal pol- icy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

2017 arXiv
[32]

Annals of operations research 134, 19–67 (2005)

De Boer, P.-T., Kroese, D.P., Mannor, S., Rubinstein, R.Y.: A tutorial on the cross-entropy method. Annals of operations research 134, 19–67 (2005)

2005
[33]

Springer (2013)

Camacho, E.F., Alba, C.B.: Model predictive control. Springer (2013)

2013
[34]

arXiv preprint arXiv:2206.15475 (2022)

Kaddour, J., Lynch, A., Liu, Q., Kusner, M.J., Silva, R.: Causal machine learning: A survey and open problems. arXiv preprint arXiv:2206.15475 (2022)

2022 arXiv
[35]

https://crl.causalai.net/

Bareinboim, E.: Towards Causal rein- forcement learning, ICML tutorial (2020). https://crl.causalai.net/

2020
[36]

IEEE Transactions on Neural Net- works and Learning Systems, 1–21 (2024) https://doi.org/10.1109/TNNLS.2024.3403001

Zeng, Y., Cai, R., Sun, F., Huang, L., Hao, Z.: A survey on causal reinforcement learning. IEEE Transactions on Neural Net- works and Learning Systems, 1–21 (2024) https://doi.org/10.1109/TNNLS.2024.3403001

2024
[37]

arXiv preprint arXiv:2111.06721 (2021)

Grimbly, S.J., Shock, J., Pretorius, A.: Causal multi-agent reinforcement learning: Review and open problems. arXiv preprint arXiv:2111.06721 (2021)

2021 arXiv
[38]

In: NeurIPS 2021 Competitions and Demonstrations Track, vol

Weichwald, S., Mogensen, S.W., Lee, T.E., Baumann, D., Kroemer, O., Guyon, I., Trimpe, S., Peters, J., Pﬁster, N.: Learning by doing: Controlling a dynamical system using causality, control, and reinforcement learning. In: NeurIPS 2021 Competitions and Demonstrations Track, vo...

2022
[39]

Advances in Neural Information Processing Systems 28 (2015)

Bareinboim, E., Forney, A., Pearl, J.: Ban- dits with unobserved confounders: A causal approach. Advances in Neural Information Processing Systems 28 (2015)

2015
[40]

The Oxford handbook of causal reasoning 1, 295 (2017)

Gershman, S.J.: Reinforcement learning and causal models. The Oxford handbook of causal reasoning 1, 295 (2017)

2017
[41]

arXiv preprint arXiv:1901.08162 (2019)

Dasgupta, I., Wang, J., Chiappa, S., Mitrovic, J., Ortega, P., Raposo, D., Hughes, E., Battaglia, P., Botvinick, M., Kurth-Nelson, Z.: Causal reasoning from meta-reinforcement learning. arXiv preprint arXiv:1901.08162 (2019)

2019 arXiv
[42]

arXiv preprint arXiv:2109.00157 (2021)

Amin, S., Gomrokchi, M., Satija, H., Hoof, H., Precup, D.: A survey of exploration meth- ods in reinforcement learning. arXiv preprint arXiv:2109.00157 (2021)

2021 arXiv
[43]

Advances in Neural Information Processing Systems 35, 20064–20076 (2022)

Peng, S., Hu, X., Zhang, R., Tang, K., Guo, J., Yi, Q., Chen, R., Zhang, X., Du, Z., Li, L., Guo, Q., Chen, Y.: Causality-driven hier- archical structure discovery for reinforcement learning. Advances in Neural Information Processing Systems 35, 20064–20076 (2022)

2022
[44]

arXiv preprint arXiv:2002.02836 (2020)

Rezende, D.J., Danihelka, I., Papamakar- ios, G., Ke, N.R., Jiang, R., Weber, T., Gregor, K., Merzic, H., Viola, F., Wang, J., et al.: Causally correct partial models for reinforcement learning. arXiv preprint arXiv:2002.02836 (2020)

2020 arXiv
[45]

Advances in Neural Information Processing Systems 33, 3976–3990 (2020)

Pitis, S., Creager, E., Garg, A.: Counterfac- tual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems 33, 3976–3990 (2020)

2020
[46]

Advances in Neural Information Processing Systems 34, 22905–22918 (2021)

Seitzer, M., Sch¨ olkopf, B., Martius, G.: Causal inﬂuence detection for improving eﬃ- ciency in reinforcement learning. Advances in Neural Information Processing Systems 34, 22905–22918 (2021)

2021
[47]

Research in Computing Science 149, 95–104 (2020)

Molina, A.M., Avelino, I.F., Morales, E.F., Sucar, L.E.: Causal based Q-learning. Research in Computing Science 149, 95–104 (2020)

2020
[48]

In: International Con- ference on Machine Learning, vol

Pathak, D., Agrawal, P., Efros, A.A., Dar- rell, T.: Curiosity-driven exploration by self- supervised prediction. In: International Con- ference on Machine Learning, vol. 70, pp. 2778–2787 (2017). PMLR

2017
[49]

In: Inter- national Conference on Learning Representa- tions (2019) 21

Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., Efros, A.A.: Large-scale study of curiosity-driven learning. In: Inter- national Conference on Learning Representa- tions (2019) 21

2019
[50]

In: Proceedings 1991 IEEE International Joint Conference on Neural Networks, pp

Schmidhuber, J.: Curious model-building control systems. In: Proceedings 1991 IEEE International Joint Conference on Neural Networks, pp. 1458–14632 (1991). https://doi.org/10.1109/IJCNN.1991.170605

1991
[51]

Advances in neural information processing systems 17 (2004)

Chentanez, N., Barto, A., Singh, S.: Intrin- sically motivated reinforcement learning. Advances in neural information processing systems 17 (2004)

2004
[52]

: Exploiting open-endedness to solve problems through the search for novelty

Lehman, J., Stanley, K.O., et al. : Exploiting open-endedness to solve problems through the search for novelty. In: ALIFE, pp. 329– 336 (2008)

2008
[53]

Frontiers in Neurorobotics 1 (2007) https://doi.org/10.3389/neuro.12.006.2007

Oudeyer, P.-Y., Kaplan, F.: What is intrinsic motivation? a typol- ogy of computational approaches. Frontiers in Neurorobotics 1 (2007) https://doi.org/10.3389/neuro.12.006.2007

2007 doi
[54]

In: Schmid- huber, J., Th´ orisson, K.R., Looks, M

Sun, Y., Gomez, F., Schmidhuber, J.: Plan- ning to be surprised: Optimal bayesian explo- ration in dynamic environments. In: Schmid- huber, J., Th´ orisson, K.R., Looks, M. (eds.) Artiﬁcial General Intelligence, pp. 41–51. Springer, Berlin, Heidelberg (2011)

2011
[55]

Theory in Biosciences 131, 139–148 (2012)

Still, S., Precup, D.: An information-theoretic approach to curiosity-driven reinforcement learning. Theory in Biosciences 131, 139–148 (2012)

2012
[56]

Springer (2013)

Baldassarre, G., Mirolli, M.: Intrinsically motivated learning in natural and artiﬁcial systems. Springer (2013)

2013
[57]

Robotics and Autonomous Systems 61(1), 49–73 (2013) https://doi.org/10.1016/j.robot.2012.05.008

Baranes, A., Oudeyer, P.-Y.: Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems 61(1), 49–73 (2013) https://doi.org/10.1016/j.robot.2012.05.008

2013 doi
[58]

(eds.) Intrinsic Motivation and Reinforcement Learning, pp

Barto, A.G.: In: Baldassarre, G., Mirolli, M. (eds.) Intrinsic Motivation and Reinforcement Learning, pp. 17–47. Springer, Berlin, Heidelberg (2013). https://doi.org/10.1007/978-3-642-32375-1 2

2013 doi
[59]

arXiv preprint arXiv:1507.00814 (2015)

Stadie, B.C., Levine, S., Abbeel, P.: Incen- tivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814 (2015)

2015 arXiv
[60]

Advances in neural information processing systems 28 (2015)

Mohamed, S., Jimenez Rezende, D.: Varia- tional information maximisation for intrin- sically motivated reinforcement learning. Advances in neural information processing systems 28 (2015)

2015
[61]

Advances in neural information pro- cessing systems 29 (2016)

Houthooft, R., Chen, X., Duan, Y., Schul- man, J., De Turck, F., Abbeel, P.: VIME: Variational information maximizing explo- ration. Advances in neural information pro- cessing systems 29 (2016)

2016
[62]

Advances in neural information processing systems 29 (2016)

Osband, I., Blundell, C., Pritzel, A., Van Roy, B.: Deep exploration via bootstrapped DQN. Advances in neural information processing systems 29 (2016)

2016
[63]

Journal of Machine Learning Research 23(152), 1–41 (2022)

Forestier, S., Portelas, R., Mollard, Y., Oudeyer, P.-Y.: Intrinsically motivated goal exploration processes with automatic curricu- lum learning. Journal of Machine Learning Research 23(152), 1–41 (2022)

2022
[64]

Advances in neural information processing systems 30 (2017)

Tang, H., Houthooft, R., Foote, D., Stooke, A., Xi Chen, O., Duan, Y., Schulman, J., DeTurck, F., Abbeel, P.: #exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems 30 (2017)

2017
[65]

In: International Conference on Machine Learn- ing, vol

Colas, C., Sigaud, O., Oudeyer, P.-Y.: Gep- pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms. In: International Conference on Machine Learn- ing, vol. 80, pp. 1039–1048 (2018). PMLR

2018
[66]

In: Conference on Robot Learning, vol

Laversanne-Finot, A., Pere, A., Oudeyer, P.- Y.: Curiosity driven exploration of learned disentangled goal spaces. In: Conference on Robot Learning, vol. 87, pp. 487–504 (2018). PMLR

2018
[67]

arXiv preprint arXiv:1802.10546 (2018)

Oudeyer, P.-Y.: Computational theories of curiosity-driven learning. arXiv preprint arXiv:1802.10546 (2018)

2018 arXiv
[68]

: BYOL-Explore: Exploration by bootstrapped prediction

Guo, Z., Thakoor, S., P ˆ ıslar, M., Avila Pires, 22 B., Altch´ e, F., Tallec, C., Saade, A., Calan- driello, D., Grill, J.-B., Tang, Y., et al. : BYOL-Explore: Exploration by bootstrapped prediction. Advances in neural information processing systems 35, 31855–31870 (2022)

2022
[69]

arXiv preprint arXiv:2106.14421 (2021)

Gasse, M., Grasset, D., Gaudron, G., Oudeyer, P.-Y.: Causal reinforcement learn- ing using observational and interventional data. arXiv preprint arXiv:2106.14421 (2021)

2021 arXiv
[70]

In: Thirty-ﬁfth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2) (2021)

Ke, N.R., Didolkar, A.R., Mittal, S., Goyal, A., Lajoie, G., Bauer, S., Rezende, D.J., Ben- gio, Y., Pal, C., Mozer, M.C.: Systematic evaluation of causal discovery in visual model based reinforcement learning. In: Thirty-ﬁfth Conference on Neural Information Process- ing Syst...

2021
[71]

arXiv preprint arXiv:1910.01751 (2019)

Nair, S., Zhu, Y., Savarese, S., Fei-Fei, L.: Causal induction from visual observa- tions for goal directed tasks. arXiv preprint arXiv:1910.01751 (2019)

2019 arXiv
[72]

Advances in Neural Information Processing Systems 35, 26532–26548 (2022)

Ding, W., Lin, H., Li, B., Zhao, D.: General- izing goal-conditioned reinforcement learning with variational causal reasoning. Advances in Neural Information Processing Systems 35, 26532–26548 (2022)

2022
[73]

arXiv preprint arXiv:1708.01289 (2017)

Thomas, V., Pondard, J., Bengio, E., Sarfati, M., Beaudoin, P., Meurs, M.-J., Pineau, J., Precup, D., Bengio, Y.: Inde- pendently controllable factors. arXiv preprint arXiv:1708.01289 (2017)

2017 arXiv
[74]

arXiv preprint arXiv:1804.03599 (2018)

Burgess, C.P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., Lerchner, A.: Understanding disentangling in beta-V AE. arXiv preprint arXiv:1804.03599 (2018)

2018 arXiv
[75]

In: International Conference on Machine Learning, vol

Kim, H., Mnih, A.: Disentangling by factoris- ing. In: International Conference on Machine Learning, vol. 80, pp. 2649–2658 (2018). PMLR

2018
[76]

Advances in neural information processing systems 31 (2018)

Chen, R.T., Li, X., Grosse, R.B., Duvenaud, D.K.: Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems 31 (2018)

2018
[77]

arXiv preprint arXiv:2002.05217 (2020)

Volodin, S., Wichers, N., Nixon, J.: Resolving spurious correlations in causal models of envi- ronments via interventions. arXiv preprint arXiv:2002.05217 (2020)

2020 arXiv
[78]

URL http://bulletphysics.org (2013)

Coumans, E., et al.: Bullet real-time physics simulation. URL http://bulletphysics.org (2013)

2013
[79]

: Triﬁnger: An open-source robot for learning dexterity

Wuthrich, M., Widmaier, F., Grimminger, F., Joshi, S., Agrawal, V., Hammoud, B., Khadiv, M., Bogdanovic, M., Berenz, V., Viereck, J., et al. : Triﬁnger: An open-source robot for learning dexterity. In: Conference on Robot Learning, vol. 155, pp. 1871–1882 (2021). PMLR

2021
[80]

MIT press (2018)

Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT press (2018)

2018
[81]

Auto- matica 14(5), 465–471 (1978) https://doi.org/10.1016/0005-1098(78)90005-5

Rissanen, J.: Modeling by shortest data description. Auto- matica 14(5), 465–471 (1978) https://doi.org/10.1016/0005-1098(78)90005-5

1978 doi
[82]

MIT Press Books 1 (2007)

Gr¨ unwald, P.D., et al.: The minimum descrip- tion length principle. MIT Press Books 1 (2007)

2007
[83]

Journal of Computational and Applied Mathematics 20, 53–65 (1987) https://doi.org/10.1016/0377-0427(87)90125-7

Rousseeuw, P.J.: Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics 20, 53–65 (1987) https://doi.org/10.1016/0377-0427(87)90125-7

1987 doi
[84]

In: International Conference on Machine Learn- ing, vol

Cuturi, M., Blondel, M.: Soft-DTW: a dif- ferentiable loss function for time-series. In: International Conference on Machine Learn- ing, vol. 70, pp. 894–903 (2017). PMLR

2017
[85]

Pattern Recognition 135, 109144 (2023) https://doi.org/10.1016/j.patcog.2022.109144

Bagirov, A.M., Aliguliyev, R.M., Sultanova, N.: Finding compact and well-separated clus- ters: Clustering using silhouette coeﬃcients. Pattern Recognition 135, 109144 (2023) https://doi.org/10.1016/j.patcog.2022.109144

2023
[86]

IEEE Access 10, 115025–115032 (2022) https://doi.org/10.1109/ACCESS.2022.3215568

Punhani, A., Faujdar, N., Mishra, K.K., Subramanian, M.: Binning-based 23 silhouette approach to ﬁnd the opti- mal cluster using K-means. IEEE Access 10, 115025–115032 (2022) https://doi.org/10.1109/ACCESS.2022.3215568

2022
[87]

Entropy 23(6) (2021) https://doi.org/10.3390/e23060759

Shutaywi, M., Kachouie, N.N.: Silhou- ette analysis for performance evaluation in machine learning with applications to clustering. Entropy 23(6) (2021) https://doi.org/10.3390/e23060759

2021 doi
[88]

Proceedings of the AAAI Conference on Artiﬁcial Intelligence 32(1) (2018) https://doi.org/10.1609/aaai.v32i1.11694

Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., Meger, D.: Deep reinforcement learning that mat- ters. Proceedings of the AAAI Conference on Artiﬁcial Intelligence 32(1) (2018) https://doi.org/10.1609/aaai.v32i1.11694

2018 doi
[89]

In: International Con- ference on Machine Learning, vol

Eimer, T., Lindauer, M., Raileanu, R.: Hyperparameters in reinforcement learning and how to tune them. In: International Con- ference on Machine Learning, vol. 202, pp. 9104–9149 (2023). PMLR

2023
[90]

arXiv preprint arXiv:2201.11182 (2022)

Kiran, M., Ozyildirim, M.: Hyperparameter tuning for deep reinforcement learning appli- cations. arXiv preprint arXiv:2201.11182 (2022)

2022 arXiv
[91]

IEEE Transac- tions on Pattern Analysis and Machine Intelligence 33(12), 2436–2450 (2011) https://doi.org/10.1109/TPAMI.2011.71

Peters, J., Janzing, D., Scholkopf, B.: Causal inference on discrete data using additive noise models. IEEE Transac- tions on Pattern Analysis and Machine Intelligence 33(12), 2436–2450 (2011) https://doi.org/10.1109/TPAMI.2011.71

2011 doi
[92]

ACM Comput

Vowels, M.J., Camgoz, N.C., Bow- den, R.: D’ya like DAGs? a survey on structure learning and causal discov- ery. ACM Comput. Surv. 55(4) (2022) https://doi.org/10.1145/3527154

2022 doi
[93]

Frontiers in Genetics 10 (2019) https://doi.org/10.3389/fgene.2019.00524

Glymour, C., Zhang, K., Spirtes, P.: Review of causal discovery methods based on graph- ical models. Frontiers in Genetics 10 (2019) https://doi.org/10.3389/fgene.2019.00524

2019
[94]

arXiv preprint arXiv:2303.15027 (2023) 24

Hasan, U., Hossain, E., Gani, M.O.: A sur- vey on causal discovery methods for tempo- ral and non-temporal data. arXiv preprint arXiv:2303.15027 (2023) 24

2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Sch¨ olkopf, B.: Causality for Machine Learning, 1st edn., pp. 765–804. Association for Computing Machin- ery, New York, NY, USA (2022). https://doi.org/10.1145/3501714.3501755

arXiv 2022

[2] [2]

Proceedings of the IEEE 109(5), 612–634 (2021) https://doi.org/10.1109/JPROC.2021.3058954

Sch¨ olkopf, B., Locatello, F., Bauer, S., Ke, N.R., Kalchbrenner, N., Goyal, A., Bengio, Y.: Toward causal rep- resentation learning. Proceedings of the IEEE 109(5), 612–634 (2021) https://doi.org/10.1109/JPROC.2021.3058954

arXiv 2021

[3] [3]

In: International Conference on Machine Learning, pp

Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch¨ olkopf, B., Bachem, O.: Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. In: International Conference on Machine Learning, pp. 4114–4124 (2019). PMLR

2019

[4] [4]

In: Interna- tional Conference on Machine Learning, pp

Suter, R., Miladinovic, D., Sch¨ olkopf, B., Bauer, S.: Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In: Interna- tional Conference on Machine Learning, pp. 6056–6065 (2019). PMLR

2019

[5] [5]

In: Proceedings of the 29th International Coference on Inter- national Conference on Machine Learning

Sch¨ olkopf, B., Janzing, D., Peters, J., Sgouritsa, E., Zhang, K., Mooij, J.: On causal and anticausal learning. In: Proceedings of the 29th International Coference on Inter- national Conference on Machine Learning. ICML’12, pp. 459–466. Omnipress, Madison, WI, USA (2012)

2012

[6] [6]

arXiv preprint arXiv:1812.00524 (2018)

Kilbertus, N., Parascandolo, G., Sch¨ olkopf, B.: Generalization in anti-causal learning. arXiv preprint arXiv:1812.00524 (2018)

arXiv 2018

[7] [7]

arXiv preprint arXiv:1812.10576 (2018)

Lu, C., Sch¨ olkopf, B., Hern´ andez-Lobato, J.M.: Deconfounding reinforcement learn- ing in observational settings. arXiv preprint arXiv:1812.10576 (2018)

arXiv 2018

[8] [8]

Singla, S., Feizi, S.: Salient imagenet: How to discover spurious features in deep learn- ing? In: International Conference on Learning Representations (2022)

2022