{"id":"379b17ca-7bde-4308-aa17-7e814607906d","arxiv_id":"2606.21173","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":8.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Value-based agents on rich reward sets implicitly encode the true transition kernel P, recoverable via P-learning under stated conditions on goals.","lead":"The paper shows that value-based RL agents trained on many different reward functions or goals implicitly learn an accurate model of the environment dynamics, which can be decoded using a new method called P-learning. A smart generalist might read it because it suggests model-free and model-based RL are more connected than thought, with potential for better generalization and model extraction without explicit modeling.","discovery_kind":"unification","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's weakest assumption (sufficiency of the chosen rewards) is directly addressed by the paper's derivation of the required conditions and by the empirical robustness results; the UNVERDICTED verdict arose solely from abstract-only access, which the full text removes without exposing a load-bearing flaw.","tokens_in":1728,"tokens_out":253,"duration_ms":24561,"concrete_test":"Re-derive the uniqueness result for a small tabular MDP (e.g., 4 states, 2 actions) by solving the system of Bellman equations for the minimal number of rewards asserted in the theorem; confirm that only the true P satisfies the system.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The manuscript supplies an explicit proof of uniqueness for the transition kernel P given Q-values over a sufficiently rich set of rewards (covering finite/continuous and deterministic/stochastic cases) together with a practical inversion procedure (P-learning) and supporting experiments. The sufficient conditions on reward type and cardinality are stated and used to ground both the theoretical claim and the empirical demonstrations; no internal gap, hidden assumption, or mismatch between the stated conditions and the reported results is apparent.","agreement_with_reader":"disagree"},"referee_report":{"model":"grok-4.3","summary":"The paper proves that value-based agents trained on a sufficiently rich set of reward functions (e.g. via goal-conditioned RL) implicitly encode a unique and accurate world model of the transition kernel P. It introduces P-learning, an inverse procedure to Q-learning that extracts the model by sampling from the agent's Q-values, policies and rewards. Sufficient conditions on the type and cardinality of goals are stated to guarantee recovery of the true P for both stochastic and deterministic MDPs over finite or continuous state spaces. Empirical results in Reacher, MountainCar and stochastic FourRooms show that agents encode accurate dynamics even when assumptions are mildly violated, and that policies trained exclusively on the implicit model achieve quasi-optimal performance on out-of-distribution velocity-based goals.","tokens_in":1813,"tokens_out":579,"duration_ms":13207,"significance":"If the central claim and proof hold, the work supplies a concrete mathematical bridge between model-free value-based RL and model-based methods by showing that sufficiently diverse reward training causes agents to contain hidden, extractable world models. The explicit uniqueness proof, the P-learning algorithm, the parameter-free character of the inversion under the stated conditions, and the generalization experiments are all strengths that would be credited in a review. The result offers a new lens on goal-conditioned RL and could motivate new algorithms for model extraction and improved OOD performance.","major_comments":[{"comment":"The uniqueness theorem (main theoretical section) asserts that Q-values over a rich reward set determine P uniquely; however, the proof sketch in the abstract and the empirical section do not clarify whether the stated cardinality conditions remain sufficient when the reward functions are linearly dependent or when the policy class is restricted, which is load-bearing for the claim that any goal-conditioned agent encodes the true kernel.","section":"theoretical results / sufficient conditions"},{"comment":"§ on P-learning: the inversion procedure is presented as sampling from Q, π and r to recover P, but the manuscript does not report the sample complexity or the numerical stability of the inversion step when Q-values are estimated from finite data; this directly affects whether the extracted model is accurate enough to support the reported quasi-optimal OOD policies.","section":"P-learning procedure"}],"minor_comments":[{"comment":"Notation for the continuous-state case should explicitly distinguish the measure-theoretic version of the Bellman equation from the finite case to avoid ambiguity in the uniqueness argument.","section":"continuous state spaces"},{"comment":"The Reacher and MountainCar experiments would benefit from an ablation that varies the number of training goals while holding total samples fixed, to quantify how quickly the implicit model accuracy saturates.","section":"empirical evaluation"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their positive assessment and recommendation of minor revision. The comments highlight important points for clarification, which we address below.","responses":[{"response":"The uniqueness theorem provides sufficient conditions on the cardinality and type of goals that ensure the reward set allows unique determination of P. These conditions are intended to guarantee that the rewards provide independent information sufficient for inversion; linear dependence among rewards would reduce the effective cardinality below the threshold, thus not satisfying the stated conditions. We will add an explicit remark in the theorem statement and surrounding discussion to clarify this aspect. The theorem applies under the policy class used by the agent for the given rewards, as detailed in the assumptions; no restriction beyond that is claimed. This clarification will be incorporated in the revision.","revision_made":"partial","referee_comment":"[theoretical results / sufficient conditions] The uniqueness theorem (main theoretical section) asserts that Q-values over a rich reward set determine P uniquely; however, the proof sketch in the abstract and the empirical section do not clarify whether the stated cardinality conditions remain sufficient when the reward functions are linearly dependent or when the policy class is restricted, which is load-bearing for the claim that any goal-conditioned agent encodes the true kernel."},{"response":"The manuscript indeed does not provide theoretical sample complexity analysis for the P-learning inversion or a dedicated study of numerical stability with finite-sample Q-value estimates. The empirical results across the environments demonstrate that the procedure yields models accurate enough for the reported OOD performance. In the revised manuscript, we will include additional discussion on the numerical stability observed in the experiments and note the lack of theoretical sample complexity bounds as a direction for future work.","revision_made":"yes","referee_comment":"[P-learning procedure] § on P-learning: the inversion procedure is presented as sampling from Q, π and r to recover P, but the manuscript does not report the sample complexity or the numerical stability of the inversion step when Q-values are estimated from finite data; this directly affects whether the extracted model is accurate enough to support the reported quasi-optimal OOD policies."}],"tokens_in":1486,"tokens_out":417,"duration_ms":30432,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper shows value-based agents trained on a sufficiently rich set of reward functions implicitly learn an accurate world model, which can be extracted via a new procedure they call P-learning. They provide proofs that the transition kernel is uniquely determined by the Q-values under certain conditions on the rewards, covering finite and continuous state spaces as well as stochastic and deterministic transitions.\n\nThe work does a good job of challenging the usual separation between model-free and model-based RL by giving explicit mathematical conditions and then demonstrating the idea empirically. The experiments on Reacher, MountainCar, and variants of FourRooms show that even with a handful of rewards the dynamics are recovered reasonably well. The additional finding that policies trained on the implicit model generalize to new goal types is an interesting side result.\n\nWhere it is softer is in the scale of the experiments. They stick to standard benchmark environments with low dimensions and use only a small number of training rewards, so the robustness to approximation errors or larger problems remains open. It would be useful to see more analysis on how the choice of rewards affects the quality of the recovered model.\n\nOverall this is for RL researchers interested in the connections between value learning, goal conditioning, and dynamics models. A reader who wants to explore whether trained agents contain hidden models would get something out of it. The formal part looks solid enough and the empirical support is there, so it deserves to go through peer review rather than being rejected at the desk.","headline":"Q-values from rich goal-conditioned training can implicitly encode the transition model, with a proof of uniqueness and a P-learning extraction method.","tokens_in":2285,"tokens_out":369,"would_cite":false,"duration_ms":21020,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Value-based agents trained on a rich set of rewards implicitly encode an accurate world model.","keywords":["reinforcement learning","world models","Q-learning","goal-conditioned RL","transition kernel","Bellman equation","model-free methods"],"falsifier":"Apply P-learning to extract a transition kernel from an agent's Q-values and compare it directly to the true environment dynamics on a held-out set of states and actions; mismatch would show the encoding claim does not hold.","tokens_in":2652,"feed_emoji":"","tokens_out":666,"duration_ms":21202,"temperature":0.7,"pith_summary":"The paper challenges the traditional split between model-free reinforcement learning, which learns values, and model-based approaches, which learn transitions. It establishes that training Q-values over enough different reward functions, as in goal-conditioned settings, builds a complete internal model of the environment's dynamics inside the agent. A new extraction method called P-learning inverts the usual process to recover that model from the agent's Q-values, policies, and rewards. If correct, this means many value-based agents already contain the information needed for planning and generalization without separate model learning.","feed_headline":"Value functions from many goals encode full dynamics","feed_subtitle":"P-learning recovers accurate transition kernels from Q-values, showing model-free agents hold hidden models for planning and generalization.","key_machinery":"P-learning, the inverse analogue to Q-learning that decodes the transition kernel from sampled Q-values, policies and rewards.","core_discovery":"Value-based agents trained on a sufficiently rich set of reward functions implicitly encode a unique and accurate world model. To extract this model in practice, P-learning samples from an agent's Q-values, policies and rewards to decode its internal model of the environment. Sufficient conditions are given on the type and number of goals for which agents encode the true transition kernel P, covering stochastic and deterministic MDPs over finite or continuous state spaces. Even when assumptions are violated, agents trained on a handful of reward functions encode accurate dynamics, and policies trained on the implicit model perform well on out-of-distribution goals.","pith_inferences":["If the encoding holds across algorithms, training on diverse goals could serve as a practical route to reliable internal simulators without explicit dynamics data.","The result offers one explanation for why goal-conditioned agents often generalize better than single-reward training.","Checking whether P-learning recovers consistent models from different value-based methods would test how general the implicit encoding is."],"forward_implications":["The extracted model from a position-only trained Reacher agent supports quasi-optimal policies on velocity-based out-of-distribution goals.","Agents encode accurate dynamics in Reacher, MountainCar and stochastic FourRooms even when the formal sufficient conditions are not fully met.","Policies trained exclusively on the agent's implicit world model match performance on tasks outside the original training distribution.","Model-free value functions contain usable transition knowledge that connects them to model-based planning."],"fun_headline_variants":["P-learning inverts Q-values to recover transition kernels","Rich goals make value functions encode full dynamics","Model-free agents hold implicit world models via rewards","Q-values from many goals decode accurate MDP kernels"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The reward functions or goals used during training must be rich enough in type and number to uniquely determine the true transition kernel.","fun_headline_variants_meta":{"raw":{"variants":["P-learning inverts Q-values to recover transition kernels","Rich goals make value functions encode full dynamics","Model-free agents hold implicit world models via rewards","Q-values from many goals decode accurate MDP kernels"]},"model":"grok-4.3","cost_usd":0.003214,"raw_usage":{"total_tokens":1760,"prompt_tokens":734,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":32137000,"prompt_tokens_details":{"text_tokens":734,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":969,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":734,"tokens_out":57,"duration_ms":7562,"temperature":1.0,"reasoning_tokens":969,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T14:31:37.639819+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Apply P-learning to extract a transition kernel from an agent's Q-values and compare it directly to the true environment dynamics on a held-out set of states and actions; mismatch would show the encoding claim does not hold.","supporting_citations":[],"review_version":1}