REVIEW 3 major objections 5 minor 23 references
Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamics
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read An agent's internal model of a continuous, partially observable control task can be recovered from its observed actions.
desk verdict A useful continuous-domain extension of inverse POMDP whose headline recovery claim is undercut by an unanalyzed hard E step. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the parameter-space control ensemble: Algorithm 1 learns $Q(b,a;\theta)$ and $\pi(a|b;\theta)$ across the whole model space $\Theta$ by repeatedly sampling parameters, simulating belief-state trajectories, and updating the value and policy networks with DDPG or Q-learning variants. Algorithm 2 then fixes those functions and performs gradient ascent on the log likelihood of the observed data, so the smooth dependence of the optimal policy on $\theta$ is what carries the inference. The extended Kalman filter supplies tractable Gaussian belief updates for the agent's posterior over states, which is what makes continuous partially observed dynamics manageable; a simplified basis for the parameter space keeps the learning tractable.
What would settle it
In a simulation where the agent's true observation trajectories are recorded, run the inference twice: once with the actual observations plugged into the likelihood, and once with only state-action data and the MAP-based hard E step. If the two parameter estimates separate systematically as the number of trajectories or their length grows, the hard E step is biased and the central recovery claim fails.
Extended reading notes
Core claim
The paper claims that inverse rational control can be scaled to POMDPs with continuous states, continuous actions, and nonlinear dynamics by making the optimal value function $Q^*(b,a;\theta)$ and optimal policy $\pi^*(a|b;\theta)$ explicit functions of the model parameters $\theta=(\theta_r,\theta_d,\theta_o)$. These generalized functions are trained once over a parameterized manifold of models, using an extended Kalman filter to represent Gaussian beliefs and deep deterministic policy gradient to optimize continuous actions. Then, for a fixed set of these functions, the likelihood of an observed state-action trajectory is maximized with respect to $\theta$, with the agent's unobserved sensory observations handled by a hard E step that substitutes their MAP estimate given the observed states. In two simulation settings, a 1D task with discrete actions and a 2D task with continuous actions, the paper reports recovery of the simulated agents' true parameters 'up to the intrinsic uncertainty,' with Fisher-information-based 95% confidence intervals covering the true values in almost all cases.
Load-bearing premise
The whole inference rests on the hard E step: replacing the agent's hidden sensory observations with their MAP estimate given the observed states must not bias the likelihood, or the recovered parameters and their confidence intervals are not trustworthy.
Editorial extensions
If this is right
- Behavioral data from continuous naturalistic tasks can be translated into quantitative estimates of an agent's subjective rewards, dynamics, and sensory noise, giving neuroscience concrete targets for what a brain might represent.
- The two-loop separation removes the nested policy optimization required by earlier inverse methods, making parameter inference feasible in continuous state-action spaces where discrete inverse POMDP methods grow intractable.
- The learned value function over parameter space can be reused for adaptive control: if task parameters change slowly, an agent can treat them as latent variables and adjust its policy without retraining from scratch.
- In real experiments, the framework could distinguish rational behavior under a wrong internal model from suboptimal behavior under the correct model, changing how apparent animal errors are interpreted.
- The same machinery can serve as a theory-of-mind component for artificial agents that must predict or coordinate with human behavior in shared environments.
Reading between the lines
- A direct extension would be to replace the hard E step with proper marginalization over the agent's observation trajectories; comparing the two on data where the true observations are known would quantify the bias, if any.
- The method's confidence intervals assume the fitted model class contains the agent's true generative process; on real biological data, model mismatch is likely to dominate, so the intervals would need recalibration through posterior predictive checks.
- If the parameter-space value function is smooth enough, the same trained ensemble could be reused across tasks that share structure, turning the upfront policy-learning cost into a reusable asset.
- A sharper test of the rationality hypothesis would be to fit behavior generated by a deliberately different, richer model and ask whether the recovered parameters still generalize to held-out actions, not just match the generating parameters.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes Inverse Rational Control (IRC), a method for estimating an agent's internal model parameters (transition, observation, and reward parameters) in a partially observable Markov decision process with continuous, nonlinear dynamics. The method first learns an ensemble of near-optimal policies and value functions over a parameterized model space using an extended Kalman filter for beliefs and DDPG/actor-critic for policy optimization, then estimates the agent's parameters by maximizing a likelihood over observable state-action trajectories. The approach is demonstrated on 1D and 2D simulated 'catching fireflies' tasks, with the claim that the true parameters of simulated agents can be recovered within Fisher-information-based confidence intervals.
Significance. If the central recovery claim survives scrutiny, this is a useful step: it extends inverse POMDP methods to continuous state and action spaces, separates expensive policy optimization from per-agent inference, and provides an interpretable framework for neuroscience applications. The policy-ensemble idea is attractive and the paper is clearly written. However, the validation is a closed-loop consistency check and the hard-E-step approximation for unobserved observations is not analyzed, so the significance is contingent on resolving the load-bearing technical concerns below.
major comments (3)
- [Section 3.2, Eq. (3) and Algorithm 2] The likelihood in Eq. (3) is written conditional on the agent's observations o_{0:T}, which the external observer does not observe. Algorithm 2 replaces these with the per-time MAP estimate \hat{o} = argmax_o O_{\theta_o}(o|s), and then optimizes the resulting complete-data likelihood. This is not a valid hard EM step: hard EM imputes o from the posterior p(o|s_{0:T},a_{0:T},\theta_old), whereas the proposed imputation uses only the observation model at each state, ignores the action sequence, and does not target the marginal likelihood p(s_{0:T},a_{0:T}|\theta). The point estimate can therefore be biased even with unlimited data if the MAP observation is inconsistent with the actions that generated the behavior, and the Fisher-information intervals computed from the imputed complete-data likelihood ignore uncertainty in the latent observations. Since the abstract and Section 4 claim recovery 'within theoretical error bounds,' this approximation is load-bearing and needs either a corrected E step, a composite-likelihood or Laplace justification, or a quantitative bound on the induced bias.
- [Section 4, Figure 1C-D] The validation is a closed-loop consistency check: the simulated agents are generated from the same parametric model class (beliefs via EKF, policies from Algorithm 1) used for inference, so the experiment tests whether the estimator recovers a member of its own search family. It does not test the method against a misspecified data-generating process, which is the situation described in the motivating neuroscience examples, where real animals are not guaranteed to fall in the model class. The paper should either add misspecification experiments (e.g., agents with wrong belief updates, bounded rationality, or different parametric assumptions) or explicitly restrict the claim to 'recovery within the model class' rather than 'the true model.' Section 5.1 lists model bias as a limitation but does not quantify or test it.
- [Section 4, confidence intervals] The reported 95% intervals are computed via 2(I^{-1/2})_{ii} from the Fisher information matrix of the imputed complete-data likelihood. No proof is given that the hard-E-step estimator attains the Cramér-Rao bound or that the intervals have nominal coverage; for estimators based on imputed data, the observed-data Fisher information (obtained by marginalizing over o) is generally smaller than the complete-data information. The paper should report empirical coverage of the intervals over repeated simulated datasets, or replace the intervals with a correctly derived observed-data information. Without this, the claim 'within theoretical error bounds' is not supported.
minor comments (5)
- [Section 4, Eq. (4)] The equation 'µt+1 = µt + gaa' appears to be missing a multiplication symbol or has a rendering typo; it should read 'g_a a_t' or similar.
- [Section 2, Relative Entropy IRL] The sentence 'The analytical solution needs a transition dynamics function which is estimated by importance sampling' is vague; it should state which quantity is estimated and by which sampler.
- [Figure 1 caption] Several panels, especially B and F, are not fully described in the caption; readers cannot tell which curves correspond to 'true' versus 'estimated' parameters or beliefs without referring to the main text.
- [Algorithm 3] Algorithm 3 is redundant with the preceding algorithms and could be removed or shortened; as written it adds little information beyond the two main algorithms.
- [Section 5.1] The limitations section should explicitly list the hard-E-step approximation as a limitation; the current list covers model bias and belief-update tractability but not the latent-observation imputation gap.
Circularity Check
No circular derivation: the fitted parameters are not defined in terms of the target quantities, and the self-citations are contextual rather than load-bearing; the hard-E-step approximation is a limitation, not a circular step.
full rationale
The paper's central inference is a maximum-likelihood fit of the agent's internal POMDP parameters from observed state-action trajectories (Eq. 3 and Algorithm 2). The objective is not constructed from the recovered parameters: the likelihood is a function of the data and candidate theta, and the optimal policy is precomputed over the parameter manifold in Algorithm 1. The sentence 'it is by construction consistent with rational behavior' explicitly acknowledges the modeling assumption rather than presenting it as an empirical discovery, so it is not a hidden circularity. The self-citations [13,22] supply the firefly task and the discrete-space predecessor framework; neither is invoked as a uniqueness theorem or as evidence that forces the numerical recovery, so they are not load-bearing. The simulations are closed-loop in the sense that the data-generating agents are drawn from the same parametric family that the method searches, but this is a standard self-consistency check, not a reduction by construction: successful recovery still depends on optimization, identifiability, and finite-sample behavior. The main statistical risk is the hard E step, which the paper itself flags ('ideally we would marginalize ... Here instead we use a hard E step'); substituting MAP observations can bias estimates and make the Fisher-information intervals undercover. That is a correctness limitation, not circularity, because the estimated parameters are not equal to the imputed observations or to the inputs by definition. No circular step can be exhibited, so the score is 0.
Assumptions & free parameters
free parameters (1)
- Uniform sampling range over model parameters Theta =
not specified ('moderate range')
assumptions (6)
- domain assumption Agents act rationally according to their own internal model of the task.
- domain assumption Agents know the basic task structure but not the task parameters.
- domain assumption Beliefs are represented by a parametric Gaussian and updated with an extended Kalman filter.
- ad hoc to paper The MAP estimate of unobserved observations is a valid substitute for marginalizing over them in the likelihood.
- ad hoc to paper The learned policy ensemble is sufficiently close to optimal across the entire parameter space.
- standard math Standard asymptotic normality of maximum likelihood estimators applies to the hard-E-step likelihood.
Cite this review
Pith. "Pith review of Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamics." pith.science (2026). https://pith.science/paper/S2SNB2LZ
@misc{pith2026190804696,
author = {Pith},
title = {Pith review of: Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamics},
year = {2026},
howpublished = {\url{https://pith.science/paper/S2SNB2LZ}},
note = {Machine review of arXiv:1908.04696}
}
read the original abstract
Continuous control and planning remains a major challenge in robotics and machine learning. Neuroscience offers the possibility of learning from animal brains that implement highly successful controllers, but it is unclear how to relate an animal's behavior to control principles. Animals may not always act optimally from the perspective of an external observer, but may still act rationally: we hypothesize that animals choose actions with highest expected future subjective value according to their own internal model of the world. Their actions thus result from solving a different optimal control problem from those on which they are evaluated in neuroscience experiments. With this assumption, we propose a novel framework of model-based inverse rational control that learns the agent's internal model that best explains their actions in a task described as a partially observable Markov decision process (POMDP). In this approach we first learn optimal policies generalized over the entire model space of dynamics and subjective rewards, using an extended Kalman filter to represent the belief space, a neural network in the actor-critic framework to optimize the policy, and a simplified basis for the parameter space. We then compute the model that maximizes the likelihood of the experimentally observable data comprising the agent's sensory observations and chosen actions. Our proposed method is able to recover the true model of simulated agents within theoretical error bounds given by limited data. We illustrate this method by applying it to a complex naturalistic task currently used in neuroscience experiments. This approach provides a foundation for interpreting the behavioral and neural dynamics of highly adapted controllers in animal brains.
Figures
Reference graph
Works this paper leans on
-
[1]
Apprenticeship learning via inverse reinforcement learning
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1. ACM, 2004. 8
work page 2004
-
[2]
Apprenticeship learning about multiple intentions
Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael L Littman. Apprenticeship learning about multiple intentions. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 897–904, 2011
work page 2011
-
[3]
Bayesian theory of mind: Modeling joint belief-desire attribution
Chris Baker, Rebecca Saxe, and Joshua Tenenbaum. Bayesian theory of mind: Modeling joint belief-desire attribution. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011
work page 2011
- [4]
-
[5]
Relative entropy inverse reinforcement learning
Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 182–189, 2011
work page 2011
-
[6]
Inverse reinforcement learning in partially observable environments
Jaedeug Choi and Kee-Eung Kim. Inverse reinforcement learning in partially observable environments. Journal of Machine Learning Research, 12(Mar):691–730, 2011
work page 2011
-
[7]
Maximum likelihood from incomplete data via the em algorithm
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–22, 1977
work page 1977
-
[8]
Inverse optimal control with linearly-solvable mdps
Krishnamurthy Dvijotham and Emanuel Todorov. Inverse optimal control with linearly-solvable mdps. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 335–342, 2010
work page 2010
Show all 23 references
-
[9]
Guided cost learning: Deep inverse optimal control via policy optimization
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pages 49–58, 2016
2016
-
[10]
Inverse reinforcement learning with simultaneous estimation of rewards and dynamics
Michael Herman, Tobias Gindele, Jörg Wagner, Felix Schmitt, and Wolfram Burgard. Inverse reinforcement learning with simultaneous estimation of rewards and dynamics. In Artificial Intelligence and Statistics, pages 102–110, 2016
2016
-
[11]
Information theory and statistical mechanics
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957
1957
-
[12]
Learning objective functions for manipulation
Mrinal Kalakrishnan, Peter Pastor, Ludovic Righetti, and Stefan Schaal. Learning objective functions for manipulation. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 1331–1336. IEEE, 2013
2013
-
[13]
A dynamic bayesian observer model reveals origins of bias in visual path integration
Kaushik J Lakshminarasimhan, Marina Petsalis, Hyeshin Park, Gregory C DeAngelis, Xaq Pitkow, and Dora E Angelaki. A dynamic bayesian observer model reveals origins of bias in visual path integration. Neuron, 2018
2018
-
[14]
Continuous control with deep reinforcement learning
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015
2015 arXiv
-
[15]
Algorithms for inverse reinforcement learning
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, pages 663–670, 2000
2000
-
[16]
Bayesian inverse reinforcement learning
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. Urbana, 51(61801):1–4, 2007
2007
-
[17]
Dragan, and Sergey Levine
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Where do you think you’re going?: Inferring beliefs about dynamics from behavior. In Arxiv 1805.08010, 2018
2018 arXiv
-
[18]
Learning agents for uncertain environments
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory, pages 101–103. ACM, 1998
1998
-
[19]
I see what you see: Inferring sensor and policy models of human real-world motor behavior
Felix Schmitt, Hans-Joachim Bieg, Michael Herman, and Constantin A Rothkopf. I see what you see: Inferring sensor and policy models of human real-world motor behavior. In AAAI, pages 3797–3803, 2017. 9
2017
-
[20]
Reinforcement learning: An introduction
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press, 1998
1998
-
[21]
Maximum likelihood inverse reinforcement learning
Monica C Vroman. Maximum likelihood inverse reinforcement learning. PhD thesis, Rutgers University-Graduate School-New Brunswick, 2014
2014
-
[22]
Inverse POMDP: Inferring what you think from what you do
Zhengwei Wu, Paul Schrater, and Xaq Pitkow. Inverse POMDP: Inferring what you think from what you do. arXiv preprint arXiv:1805.09864, 2018
2018 arXiv
-
[23]
Maximum entropy inverse reinforcement learning
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 1433–1438. Chicago, IL, USA, 2008. 10
2008
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.