{"id":"d2563908-613d-4e38-847d-a2fe980f8d68","arxiv_id":"1908.04696","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A framework that recovers an agent's assumed dynamics, rewards, and observation noise in continuous partially observable tasks by training policies over a model space and maximizing the likelihood of observed actions.","lead":"The authors propose a method to infer an animal's internal model of the world from its actions in continuous control tasks. If it works, it could let neuroscientists measure subjective beliefs and rewards from behavior.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hard E step in Algorithm 2 substitutes a MAP observation into the complete-data likelihood (Eq. 3), so the reported Fisher-information intervals can understate uncertainty; the central recovery claim is not yet supported without an analysis of this approximation.","rationale":"The reader's weakest assumption and my analysis converge on the hard E step in Section 3.2. The central claim of the paper is that Algorithm 2 recovers true parameters within theoretical error bounds, and that claim depends on the likelihood and its curvature being the right inferential objects. Eq. 3 is explicitly conditioned on observations the observer does not have, and the paper's replacement with a MAP observation is acknowledged but not analyzed. Moreover, the MAP step uses only the state, not the action sequence, so it is not a standard hard EM posterior mode given the observed data. This can bias point estimates and, more directly, make the Fisher-information intervals overconfident because they treat imputed observations as known. The paper's own Limitations section names model bias and belief-update tractability but does not address this approximation, which is the most load-bearing gap for the recovery claim. I do not recommend rejection because the concern is concrete and testable: a simulation comparing the current estimator, a proper hard EM variant, and a marginal-likelihood estimator would settle whether the approximation is harmful in the demonstrated task. Until then, the paper should be conditional on such a check or an analytic bound. Other candidate concerns, such as the policy ensemble being not fully optimal or parameter identifiability, are real but secondary: the hard E step directly threatens the validity of the reported error bars.","tokens_in":8750,"tokens_out":5596,"duration_ms":60511,"concrete_test":"Simulate the 2D firefly task with a known \\theta*, generating many trajectories. Estimate \\theta three ways: (a) Algorithm 2 exactly as implemented; (b) a proper hard EM step that imputes observations from p(o|s,a,\\theta_old); (c) an approximate marginal likelihood over observations using a particle filter, or, if needed, a simplified discrete-observation version of the task. Compare biases and empirical coverage of the nominal 95% Fisher intervals across the three estimators. If Algorithm 2's coverage is materially below 95% or its bias is not explained by finite sample size, the concern lands; if it matches (c), the hard E step is not the bottleneck.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.2 defines the log likelihood in Eq. 3 conditional on the agent's observations, but the observer has only states and actions. Algorithm 2 replaces the unobserved observations by \\hat{o} = argmax_o O_{\\theta_o}(o|s) and then treats the result as if it were the likelihood. This is not a valid hard EM step: the imputed observations are not drawn from the posterior p(o|s,a,\\theta_old), they ignore the action sequence, and the expression maximized is not the marginal likelihood p(s_{0:T}, a_{0:T}|\\theta). Two consequences follow. First, the point estimate can be biased even with abundant data if the MAP observation is inconsistent with the actions that produced the behavior. Second, the Fisher information computed from the imputed complete-data likelihood ignores uncertainty in the latent observations, so the 95% intervals in Figure 1D can substantially undercover. The paper provides no bound on this approximation; the Limitations section (5.1) lists model bias and belief-update tractability but not this hard-E-step gap. Because the headline claim is recovery 'within theoretical error bounds', this unanalyzed approximation is the load-bearing concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Inverse Rational Control (IRC), a method for estimating an agent's internal model parameters (transition, observation, and reward parameters) in a partially observable Markov decision process with continuous, nonlinear dynamics. The method first learns an ensemble of near-optimal policies and value functions over a parameterized model space using an extended Kalman filter for beliefs and DDPG/actor-critic for policy optimization, then estimates the agent's parameters by maximizing a likelihood over observable state-action trajectories. The approach is demonstrated on 1D and 2D simulated 'catching fireflies' tasks, with the claim that the true parameters of simulated agents can be recovered within Fisher-information-based confidence intervals.","tokens_in":8999,"tokens_out":4651,"duration_ms":45069,"significance":"If the central recovery claim survives scrutiny, this is a useful step: it extends inverse POMDP methods to continuous state and action spaces, separates expensive policy optimization from per-agent inference, and provides an interpretable framework for neuroscience applications. The policy-ensemble idea is attractive and the paper is clearly written. However, the validation is a closed-loop consistency check and the hard-E-step approximation for unobserved observations is not analyzed, so the significance is contingent on resolving the load-bearing technical concerns below.","major_comments":[{"comment":"The likelihood in Eq. (3) is written conditional on the agent's observations o_{0:T}, which the external observer does not observe. Algorithm 2 replaces these with the per-time MAP estimate \\hat{o} = argmax_o O_{\\theta_o}(o|s), and then optimizes the resulting complete-data likelihood. This is not a valid hard EM step: hard EM imputes o from the posterior p(o|s_{0:T},a_{0:T},\\theta_old), whereas the proposed imputation uses only the observation model at each state, ignores the action sequence, and does not target the marginal likelihood p(s_{0:T},a_{0:T}|\\theta). The point estimate can therefore be biased even with unlimited data if the MAP observation is inconsistent with the actions that generated the behavior, and the Fisher-information intervals computed from the imputed complete-data likelihood ignore uncertainty in the latent observations. Since the abstract and Section 4 claim recovery 'within theoretical error bounds,' this approximation is load-bearing and needs either a corrected E step, a composite-likelihood or Laplace justification, or a quantitative bound on the induced bias.","section":"Section 3.2, Eq. (3) and Algorithm 2"},{"comment":"The validation is a closed-loop consistency check: the simulated agents are generated from the same parametric model class (beliefs via EKF, policies from Algorithm 1) used for inference, so the experiment tests whether the estimator recovers a member of its own search family. It does not test the method against a misspecified data-generating process, which is the situation described in the motivating neuroscience examples, where real animals are not guaranteed to fall in the model class. The paper should either add misspecification experiments (e.g., agents with wrong belief updates, bounded rationality, or different parametric assumptions) or explicitly restrict the claim to 'recovery within the model class' rather than 'the true model.' Section 5.1 lists model bias as a limitation but does not quantify or test it.","section":"Section 4, Figure 1C-D"},{"comment":"The reported 95% intervals are computed via 2(I^{-1/2})_{ii} from the Fisher information matrix of the imputed complete-data likelihood. No proof is given that the hard-E-step estimator attains the Cramér-Rao bound or that the intervals have nominal coverage; for estimators based on imputed data, the observed-data Fisher information (obtained by marginalizing over o) is generally smaller than the complete-data information. The paper should report empirical coverage of the intervals over repeated simulated datasets, or replace the intervals with a correctly derived observed-data information. Without this, the claim 'within theoretical error bounds' is not supported.","section":"Section 4, confidence intervals"}],"minor_comments":[{"comment":"The equation 'µt+1 = µt + gaa' appears to be missing a multiplication symbol or has a rendering typo; it should read 'g_a a_t' or similar.","section":"Section 4, Eq. (4)"},{"comment":"The sentence 'The analytical solution needs a transition dynamics function which is estimated by importance sampling' is vague; it should state which quantity is estimated and by which sampler.","section":"Section 2, Relative Entropy IRL"},{"comment":"Several panels, especially B and F, are not fully described in the caption; readers cannot tell which curves correspond to 'true' versus 'estimated' parameters or beliefs without referring to the main text.","section":"Figure 1 caption"},{"comment":"Algorithm 3 is redundant with the preceding algorithms and could be removed or shortened; as written it adds little information beyond the two main algorithms.","section":"Algorithm 3"},{"comment":"The limitations section should explicitly list the hard-E-step approximation as a limitation; the current list covers model bias and belief-update tractability but not the latent-observation imputation gap.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a preprint under review; the contribution relative to the authors' earlier inverse POMDP work [22] is incremental but real. The main risk is the hard-E-step approximation; if the authors cannot fix it, the central claim should be weakened. The closed-loop validation is a further limitation that should be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real step forward for inverse POMDP methods in continuous domains, but the headline claim of recovery within theoretical error bounds is not yet supported because the hard E step that imputes missing observations is unanalyzed and likely biases the Fisher-information intervals.\n\nWhat's new: the authors extend their earlier discrete-state inverse POMDP to continuous states, continuous actions, and nonlinear dynamics by using an extended Kalman filter for beliefs, a DDPG-trained actor-critic to learn optimal policies across a parameterized family of POMDPs, and gradient-based MLE for the agent's model parameters. The two-stage separation of policy learning and parameter inference is a sensible way around the nested-loop cost of classic IRL. The demonstration on the firefly task is well chosen and the simulated recovery plots are convincing as a sanity check. The citation pattern is straightforward; the self-citations to [13,22] point to the authors' own relevant prior work, which is legitimate.\n\nWhere it gets shaky: the likelihood in Eq. 3 is conditioned on the agent's observations, which the observer does not have. Algorithm 2 replaces them with the MAP estimate given the observed state, calls it a hard E step, and then treats the complete-data likelihood as the objective. That is not an EM step: the imputed observations are not drawn from the posterior given the actions and current parameters, and the action sequence is ignored in the imputation. The Fisher information computed from this imputed likelihood therefore has no guarantee of covering the true parameters; the error bars in Fig. 1 could understate uncertainty by a lot. The reader's other concerns are fair but secondary: the simulations use the same parametric model class for generation and inference, so they test internal consistency rather than robustness, and no code/data are provided to check the figures.\n\nThe Limitations section lists model bias and belief-update tractability but does not mention this hard-E-step gap, which is the load-bearing issue. If the authors can bound the approximation error, replace the imputation with a proper Monte Carlo E step, or at least validate coverage on simulated data where the true observations are known, the paper becomes much stronger.\n\nWho it's for: computational neuroscientists who want to estimate animals' subjective dynamics, rewards, and sensory noise in continuous naturalistic tasks. With that technical fix and a code release, it deserves a serious referee. I'd accept it for review but flag the hard E step as a required revision. I wouldn't cite it yet until the approximation is handled.","headline":"A useful continuous-domain extension of inverse POMDP whose headline recovery claim is undercut by an unanalyzed hard E step.","tokens_in":9485,"tokens_out":2844,"would_cite":false,"duration_ms":30101,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An agent's internal model of a continuous, partially observable control task can be recovered from its observed actions.","keywords":["inverse rational control","inverse reinforcement learning","partially observable Markov decision process","continuous control","belief space","extended Kalman filter","actor-critic","parameter space"],"falsifier":"In a simulation where the agent's true observation trajectories are recorded, run the inference twice: once with the actual observations plugged into the likelihood, and once with only state-action data and the MAP-based hard E step. If the two parameter estimates separate systematically as the number of trajectories or their length grows, the hard E step is biased and the central recovery claim fails.","tokens_in":8547,"feed_emoji":"🧠","tokens_out":6561,"duration_ms":65336,"temperature":0.7,"pith_summary":"This paper sets out to show that an agent's internal model of a partially observable continuous control task—its assumed dynamics, reward function, and sensory noise—can be recovered from externally visible state-action trajectories. The motivating hypothesis is that animals act rationally with respect to their own subjective model of the world, not optimally with respect to the experimenter's task, so their choices are evidence about that internal model. The method first learns optimal policies generalized across the entire parameter space of models, then performs gradient-based maximum likelihood estimation to find which model best explains observed behavior. On simulated firefly-catching tasks, the recovered parameters match the true parameters within confidence intervals derived from limited data. If it holds beyond simulation, this would give neuroscience a way to read out the beliefs and assumptions that drive animal behavior.","feed_headline":"Tool recovers an agent's assumed world model from its actions alone","feed_subtitle":"For continuous, partially observable control, it fits the dynamics, reward, and sensory-noise parameters an agent acts on.","key_machinery":"The load-bearing object is the parameter-space control ensemble: Algorithm 1 learns $Q(b,a;\\theta)$ and $\\pi(a|b;\\theta)$ across the whole model space $\\Theta$ by repeatedly sampling parameters, simulating belief-state trajectories, and updating the value and policy networks with DDPG or Q-learning variants. Algorithm 2 then fixes those functions and performs gradient ascent on the log likelihood of the observed data, so the smooth dependence of the optimal policy on $\\theta$ is what carries the inference. The extended Kalman filter supplies tractable Gaussian belief updates for the agent's posterior over states, which is what makes continuous partially observed dynamics manageable; a simplified basis for the parameter space keeps the learning tractable.","core_discovery":"The paper claims that inverse rational control can be scaled to POMDPs with continuous states, continuous actions, and nonlinear dynamics by making the optimal value function $Q^*(b,a;\\theta)$ and optimal policy $\\pi^*(a|b;\\theta)$ explicit functions of the model parameters $\\theta=(\\theta_r,\\theta_d,\\theta_o)$. These generalized functions are trained once over a parameterized manifold of models, using an extended Kalman filter to represent Gaussian beliefs and deep deterministic policy gradient to optimize continuous actions. Then, for a fixed set of these functions, the likelihood of an observed state-action trajectory is maximized with respect to $\\theta$, with the agent's unobserved sensory observations handled by a hard E step that substitutes their MAP estimate given the observed states. In two simulation settings, a 1D task with discrete actions and a 2D task with continuous actions, the paper reports recovery of the simulated agents' true parameters 'up to the intrinsic uncertainty,' with Fisher-information-based 95% confidence intervals covering the true values in almost all cases.","pith_inferences":["A direct extension would be to replace the hard E step with proper marginalization over the agent's observation trajectories; comparing the two on data where the true observations are known would quantify the bias, if any.","The method's confidence intervals assume the fitted model class contains the agent's true generative process; on real biological data, model mismatch is likely to dominate, so the intervals would need recalibration through posterior predictive checks.","If the parameter-space value function is smooth enough, the same trained ensemble could be reused across tasks that share structure, turning the upfront policy-learning cost into a reusable asset.","A sharper test of the rationality hypothesis would be to fit behavior generated by a deliberately different, richer model and ask whether the recovered parameters still generalize to held-out actions, not just match the generating parameters."],"forward_implications":["Behavioral data from continuous naturalistic tasks can be translated into quantitative estimates of an agent's subjective rewards, dynamics, and sensory noise, giving neuroscience concrete targets for what a brain might represent.","The two-loop separation removes the nested policy optimization required by earlier inverse methods, making parameter inference feasible in continuous state-action spaces where discrete inverse POMDP methods grow intractable.","The learned value function over parameter space can be reused for adaptive control: if task parameters change slowly, an agent can treat them as latent variables and adjust its policy without retraining from scratch.","In real experiments, the framework could distinguish rational behavior under a wrong internal model from suboptimal behavior under the correct model, changing how apparent animal errors are interpreted.","The same machinery can serve as a theory-of-mind component for artificial agents that must predict or coordinate with human behavior in shared environments."],"supporting_citations":[{"why":"poses the inverse POMDP problem in discrete state spaces with discretized beliefs, the approach this paper extends to continuous states and controls","marker":"[22]"},{"why":"provides the firefly-catching task and the dynamic Bayesian observer model used as the demonstration setting and as the source of behavioral biases","marker":"[13]"},{"why":"supplies the Deep Deterministic Policy Gradient algorithm used to learn continuous policies and value functions over parameter space in Algorithm 1","marker":"[14]"},{"why":"defines the EM framework that motivates the hard E step used to handle the agent's unobserved observations in Algorithm 2","marker":"[7]"},{"why":"gives the POMDP formulation, Bellman equation, and action-selection background that the inverse rational control problem is built on","marker":"[20]"},{"why":"infers beliefs about dynamics from behavior in a fully observable setting, marking the boundary this paper crosses into partial observability","marker":"[17]"},{"why":"introduces dynamic programming and the Bellman equation that underpin the value function and policy optimization in the ensemble","marker":"[4]"}],"fun_headline_variants":["From actions, infer the agent's world model","Inverse control infers an agent's assumed world","Actions alone reveal an agent's assumed world model","Recover the world model an agent acts on","Uncover the internal model dictating an agent's actions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole inference rests on the hard E step: replacing the agent's hidden sensory observations with their MAP estimate given the observed states must not bias the likelihood, or the recovered parameters and their confidence intervals are not trustworthy.","fun_headline_variants_meta":{"raw":{"variants":["From actions, infer the agent's world model","Inverse control infers an agent's assumed world","Actions alone reveal an agent's assumed world model","Recover the world model an agent acts on","Uncover the internal model dictating an agent's actions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000757,"raw_usage":{"total_tokens":3401,"prompt_tokens":1020,"completion_tokens":2381,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":2307}},"tokens_in":636,"tokens_out":2381,"duration_ms":16328,"temperature":1.0,"reasoning_tokens":2307,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:34:55.656529+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a simulation where the agent's true observation trajectories are recorded, run the inference twice: once with the actual observations plugged into the likelihood, and once with only state-action data and the MAP-based hard E step. If the two parameter estimates separate systematically as the number of trajectories or their length grows, the hard E step is biased and the central recovery claim fails.","supporting_citations":[{"cited_title":"Inverse Rational Control: Inferring What You Think from How You Forage","cited_arxiv_id":"1805.09864","evidence_quote":"poses the inverse POMDP problem in discrete state spaces with discretized beliefs, the approach this paper extends to continuous states and controls"},{"cited_title":"A dynamic bayesian observer model reveals origins of bias in visual path integration","cited_arxiv_id":null,"evidence_quote":"provides the firefly-catching task and the dynamic Bayesian observer model used as the demonstration setting and as the source of behavioral biases"},{"cited_title":"Maximum likelihood from incomplete data via the em algorithm","cited_arxiv_id":null,"evidence_quote":"defines the EM framework that motivates the hard E step used to handle the agent's unobserved observations in Algorithm 2"},{"cited_title":"Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior","cited_arxiv_id":"1805.08010","evidence_quote":"infers beliefs about dynamics from behavior in a fully observable setting, marking the boundary this paper crosses into partial observability"},{"cited_title":"Dynamic programming","cited_arxiv_id":null,"evidence_quote":"introduces dynamic programming and the Bellman equation that underpin the value function and policy optimization in the ensemble"}],"review_version":1}