{"id":"61a56110-202e-4667-b5f1-65c200fb3393","arxiv_id":"2507.02639","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Information-gain exploration bonuses are claimed to converge and to make model-based exploration sample-efficient, but the central mathematical proof does not hold together.","lead":"This paper proposes PTS-BE, a model-based reinforcement learning framework that combines information-gain exploration bonuses with predictive trajectory sampling. The advertised formal guarantees for these bonuses contain a critical proof gap, even though the empirical comparisons show large sample-efficiency gains on sparse-reward tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 3.1 is unproven: the proof evaluates H[δθ0]−H[δθ0]=0 although differential entropy is undefined at a point mass, and no continuity argument transfers posterior consistency (or the ε_n contraction rate) to the information-gain functional.","rationale":"The paper has two distinct contributions: a set of information-theoretic identities and an engineering framework (PTS-BE), and a claimed formal proof that IG bonuses vanish. The first part is sound: Eqs. (4)–(6) are standard EIG/JSD identities and the deep-ensemble approximation is reasonable. The empirical plots show that PTS-BE-EIG explores more efficiently than baselines in the tested environments. The load-bearing problem is the second part. The proof of Proposition 3.1 in Appendix A.1 is not a minor gap: it replaces an undefined limit H[δθ0]−H[δθ0] by 0 and invokes Proposition A.2 for a functional IG_n that is not a fixed continuous map of a consistent estimator. These are the only arguments supporting the self-terminating-bonus claim, the value-function corollary, and the claimed minimax rate. Without them, the central advertised guarantee is unproven, even if the empirical behavior in Fig. 2 suggests the conclusion may hold in practice. I agree with the Reader's weakest-assumption analysis and see no reason to alter the REJECT verdict; a revision would need a new, finite-n proof of IG convergence and a rate-transfer lemma before the theoretical claims could be accepted.","tokens_in":31862,"tokens_out":10310,"duration_ms":133453,"concrete_test":"Analytically verify the transfer step in the simplest case of Eq. (3): θ∼N(0,1), y_i|θ∼N(θ,1). The exact expected information gain for the next observation is (1/2)log(1+1/(n+1)), which tends to zero at rate 1/n; this is obtained from finite-n expectations, never from H[δθ0]−H[δθ0]. Since the appendix's step cannot reproduce this calculation, the proof is invalid as written. To settle the general claim, attempt to prove Proposition 3.1 by bounding IG_n = E[KL(p(y|s,a,D_n)∥p(y|s,a))] using dominated convergence and an integrability condition on the posterior predictive densities; if that derivation cannot be completed without extra assumptions not stated in the paper, the advertised guarantee is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 3.1 and its corollaries are the advertised formal guarantee that information-gain bonuses self-terminate. The proof in Appendix A.1 does not establish this. It first invokes Schwartz posterior consistency to get π(θ|ξ_t) ⇒ δθ0, then asserts that a new datum adds no information at the limit and concludes IGθ → H[δθ0]−H[δθ0]=0. That limit is invalid: differential entropy of a point mass is undefined (−∞ in the usual convention), so the expression is an ∞−∞ indeterminate form, not a number. More importantly, mutual information and entropy differences are not continuous functionals under weak convergence; posterior consistency does not automatically imply convergence of the finite-sample information gain. A second concrete defect is that the proof tries to apply Proposition A.2 by taking g(θ̂_n)=IGθ(ξ_n,s_{t+1}), but IG_n is not a fixed continuous function of a point estimator θ̂_n: it depends on the whole posterior and on the conditioning observation, so consistency of θ̂_n does not transfer to convergence of IG_n. The same gap invalidates Proposition 3.3 and Corollary A.5: Corollary A.7 transfers the contraction rate ε_n to IG and to the value functions without a modulus-of-continuity bound for those functionals. The empirical decay in Fig. 2 is suggestive, but the paper's central theoretical claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies information-theoretic intrinsic rewards for model-based reinforcement learning. It defines information gain IG_θ(s,a,s') and an expected information gain EIG_θ(s,a), derives the identity EIG = Jensen-Shannon divergence, proposes the PTS-BE planning-to-explore algorithm with GP/Deep Kernel/Deep Ensemble dynamics models, and reports experiments in Mountain Car, unichain, and maze environments. The paper's central theoretical contribution is Proposition 3.1 (IG converges to zero under posterior consistency), Corollary 3.2 (BAMDP value functions converge to the original MDP value functions), and Proposition 3.3 (IG contracts at the minimax rate ε_n = n^{-1/(2+|X|/α)}). I find the EIG=JSD derivation in Eq. (6) to be correct and standard, and the empirical study is reasonably broad, but the proof of the advertised convergence and rate results is invalid as written. The central formal claim that information-gain bonuses self-terminate is therefore not established.","tokens_in":32200,"tokens_out":6461,"duration_ms":76139,"significance":"If the convergence and rate results were valid, they would provide a principled, self-terminating exploration bonus with quantitative decay guarantees, complementing earlier work such as VIME, Plan2Explore, and Model-Based Active Exploration. The derivation that expected information gain equals a Jensen-Shannon divergence among ensemble members is a useful and correct simplification, and the empirical framework is plausible. However, the paper's headline theoretical contribution rests on a proof that treats the differential entropy of a point mass as zero and asserts without proof that information-gain functionals inherit posterior contraction rates. These are load-bearing gaps, not presentation issues. As a result, the theoretical significance of the paper is substantially reduced, although the empirical comparisons remain suggestive.","major_comments":[{"comment":"The proof concludes by writing IG_{θ0}(ξ_n, s_{t+1}) = H[δθ0] − H[δθ0] = 0. This is mathematically invalid: differential entropy of a Dirac point mass is undefined (and in the usual convention is −∞), so the expression is an indeterminate form. Posterior consistency in the weak topology does not imply convergence of differential entropies or of mutual information, and no continuity argument is supplied. A separate argument is needed to show that H[p(θ|ξ_t)] − H[p(θ|ξ_t, s_{t+1})] converges to zero. Without such an argument, Proposition 3.1 is unproven.","section":"Appendix A.1, proof of Proposition 3.1"},{"comment":"The proof states that one can pick g(θ̂_n) to be exactly T(ξ_n) = IG_θ(ξ_n, s_{t+1}) and then apply Proposition A.2. This is not a valid application: Proposition A.2 concerns a fixed continuous function g of a point estimator θ̂_n, whereas IG_θ(ξ_n, s_{t+1}) is a functional of the entire posterior distribution and depends on the conditioning observation s_{t+1}. Consistency of a point estimator therefore does not transfer to convergence of the information gain. This invalidates the central step of the proof of Proposition 3.1.","section":"Appendix A.1, use of Proposition A.2"},{"comment":"The rate claim is asserted rather than proved. After invoking posterior contraction at rate ε_n, the text states that IG_θ(ξ_n, s_{t+1}) → 0 and V^{augm} → V^* at the same rate ε_n. Corollary A.7 concerns estimators, not functionals of the posterior, and no modulus-of-continuity bound is given for the information-gain functional or for the value functions. The sentence in Appendix A.2 that 'posterior consistency implies the existence of a contraction rate... IG → 0 at the same rate' is exactly the missing mathematical step. The claimed minimax rate for information gain is therefore unsupported.","section":"Appendix A.2 and Proposition 3.3"},{"comment":"Convergence of the BAMDP value function to the original MDP value function requires that the intrinsic reward vanishes uniformly over the state-action space, or at least uniformly along the state-action pairs encountered under the policy π. The proof only attempts to establish pointwise convergence of IG for a single transition (s,a,s'), and then asserts convergence of V^{augm} and Q^{augm}. No argument is given to control the Bellman recursion uniformly. Consequently, Corollary 3.2 and Corollary A.5 do not follow from the preceding pointwise consistency claim.","section":"Corollary 3.2 and Corollary A.5"},{"comment":"The formal results concern the realized information gain IG_θ(s,a,s'), whereas the PTS-BE algorithm and the empirical validation use the expected information gain EIG_θ(s,a). No theorem states that EIG_θ(s,a) converges to zero under posterior consistency, and the EIG=JSD identity of Eq. (6) by itself does not provide such a rate. Figure 2 is an empirical illustration, not a proof. Thus the theoretical guarantees advertised in the abstract do not cover the exploration signal actually used in the proposed framework.","section":"Sections 3.2 and 4 (theory versus algorithm)"}],"minor_comments":[{"comment":"The text refers to 'Proposition 2.1' when stating posterior consistency; this should be Definition A.1 or Theorem A.4. The notation ξ_n^t versus ξ_t^n is also inconsistent.","section":"Appendix A.1"},{"comment":"The main text says an additional experiment on a noisy L=100 unichain environment is included, but Appendix D.2 describes the L=100 unichain as 'without added noise this time'. This is a factual inconsistency that should be corrected.","section":"Section 5.2 and Appendix D.2"},{"comment":"Figure 2 reports uncertainty measures without error bars or replication details, so the empirical decay of EIG is illustrative rather than quantitatively established. Specifying the number of runs and the variance across runs would strengthen the claim.","section":"Figure 2"},{"comment":"In the first displayed equation of the EIG-as-disagreement derivation, the integral is written as ∫_Θ p(s_{t+1}|s_t,a_t;θ) dθ, omitting the posterior weight p(θ|s_t,a_t) inside the integral; the correct expression appears in Eq. (6) of the main text.","section":"Appendix B.3"},{"comment":"There are several typographical errors, including 'convergences' in Section 3.2, 'emplyed' in the Figure 10 caption, 'enviroment' in Appendix D.1, and 'performance' instead of 'perform' in Section 5.1.1. A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript's central theoretical claims fail at the proof level, and the gap is not a routine patch: it requires new regularity conditions under which information gain is continuous in the posterior and under which contraction rates transfer from posterior distributions to entropy/mutual-information functionals. The correct EIG=JSD derivation and the broad empirical study do not rescue the advertised formal contribution. I recommend rejection, though a substantially revised manuscript that either proves convergence under explicit assumptions or reframes the contribution as primarily empirical could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ignore the advertised theorem, take the empirical package. The paper's central claim—that information-gain bonuses provably converge to zero under posterior consistency—is not proven. Appendix A.1 evaluates H[δθ0]−H[δθ0]=0, which is an undefined ∞−∞ expression, and the step from posterior consistency to convergence of IG relies on a continuity that is not established and is not standard for differential entropy. Corollary A.7 transfers contraction rates to IG and value functions without a modulus of continuity. The stress-test note is right; the reader's REJECT is the correct verdict on the theoretical core.\n\nThat said, the paper does real work. The EIG=JSD derivation (Eq. 6 and Appendix B.3) is correct and clean; it's a known identity but well presented. The PTS-BE framework is a sensible generalization of planning-to-explore (Shyam et al. 2019, Sekar et al. 2020) to arbitrary Bayesian dynamics models, and the empirical comparison across Mountain Car, unichain, and maze environments is more thorough than most. Figure 2's decay of EIG versus plateauing prediction error is a nice empirical illustration, and the heteroskedastic noise discussion in Section 4.1 is well-motivated. The authors are candid that VIME and planning-to-explore lack theory, and they position their contribution accurately.\n\nThe soft spots beyond the proof: the rate claim in Prop 3.3 is imported from van der Vaart–van Zanten, so it is only as new as the transfer argument, which fails. The value-function convergence corollary inherits the same gap. The free parameters (η_t, horizon J, rollouts K, Matérn smoothness) are not tuned systematically, but that is normal for this literature. Minor: the noisy TV discussion is fine but not deep.\n\nWho gets value: people building exploration bonuses for MBRL can use the empirical comparison and the EIG=JSD equivalence as a recipe. People looking for a rigorous guarantee that IG bonuses self-terminate should not rely on this paper. It deserves a serious referee—the empirical work is substantial and the theoretical failure is instructive—but as it stands the central advertised result is unsupported. I would not cite it for the theorem.","headline":"Useful empirical package and a correct EIG=JSD identity, but the advertised convergence theorem rests on an invalid entropy-limit argument; the theoretical centerpiece does not hold.","tokens_in":32720,"tokens_out":1907,"would_cite":false,"duration_ms":22905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that information-gain intrinsic rewards—Bayesian surprise bonuses—converge to zero as the agent's posterior tightens, so the Bayes-adaptive value function converges back to the original MDP's value function.","keywords":["information gain","intrinsic motivation","exploration bonus","Bayesian reinforcement learning","model-based reinforcement learning","epistemic uncertainty","deep ensembles","posterior consistency"],"falsifier":"Construct a consistent posterior sequence that converges weakly to a Dirac measure but whose differential entropy does not converge to the entropy of a point mass—for example a Gaussian posterior with variance $1/n$, whose differential entropy grows like $\\tfrac12\\log(2\\pi e/n) \\to -\\infty$—and compute $H[p(\\theta|D_n)] - H[p(\\theta|D_n,s')]$ after one new observation; if this difference does not tend to zero while the posterior is consistent, the claimed implication in Proposition 3.1 fails.","tokens_in":31642,"feed_emoji":"🧭","tokens_out":6068,"duration_ms":68456,"temperature":0.7,"pith_summary":"The paper studies exploration bonuses defined as Bayesian surprise—the information gain about unknown transition dynamics from observing a new transition. It claims that under standard posterior consistency, the one-step information gain $IG_\\theta(s,a,s')$ converges to zero in probability as the number of transitions grows, and consequently the value function of the Bayes-adaptive MDP converges to the value function of the original MDP. This would make information-gain exploration self-terminating: the bonus is a genuine signal of epistemic uncertainty that fades exactly as the agent becomes certain. The paper also shows that for deep ensembles, the expected information gain reduces to the Jensen–Shannon divergence between the ensemble members' predictive distributions, and it packages this into a planning algorithm, PTS-BE, that outperforms reactive intrinsic-reward and model-free baselines in sparse-reward and pure-exploration environments.","feed_headline":"Proof: information-gain exploration bonuses fade to zero","feed_subtitle":"Bayesian value functions converge back to the original MDP, so curiosity-driven exploration self-terminates.","key_machinery":"The central object is the information gain $IG_\\theta(s,a,s') = H[p(\\theta|s,a)] - H[p(\\theta|s,a,s')]$, the reduction in Shannon entropy of the dynamics-parameter posterior after a transition is observed. The convergence argument routes this through posterior consistency: the posterior contracts to the Dirac measure $\\delta_{\\theta_0}$, and the paper treats the differential entropy of that limiting point mass as zero, giving $IG \\to 0$. For planning, the paper uses the predictive surrogate $EIG_\\theta(s,a) = H[p(s'|s,a)] - \\mathbb{E}_{\\theta|s,a}[H[p(s'|s,a;\\theta)]]$, which for an ensemble of probabilistic dynamics models equals the Jensen–Shannon divergence among the ensemble members' predictive distributions—so the exploration bonus can be computed by disagreement before the next state or reward is observed.","core_discovery":"At the center of the paper is a proof that information-theoretic intrinsic rewards vanish exactly when they should. For transition dynamics with additive Gaussian noise, a prior on the function $f$ and noise variance $\\sigma^2$, and data satisfying Schwartz's posterior-consistency conditions, the one-step information gain $IG_\\theta(s,a,s')$ tends to zero in probability as $n$ grows. The paper's corollary extends this to the value level: the Bayes-adaptive MDP value function, defined with augmented rewards $r = r_e + \\eta_i IG_\\theta$, converges in probability to the original MDP's value function, so the exploration signal does not permanently bias the learned policy. Under a Matérn Gaussian-process prior on a smooth transition function, the same convergence is claimed to occur at the minimax posterior contraction rate $n^{-1/(2+|\\mathcal{X}|/\\alpha)}$.","pith_inferences":["If the self-termination theorem holds only under added continuity conditions on differential entropy, then practical guarantees would require verifying those conditions or replacing entropy differences with a divergence that is continuous under weak convergence.","EIG-as-disagreement inherits the quality of the ensemble posterior: with few or poorly diversified members, disagreement can vanish before genuine knowledge is complete, so ensemble diversity is a testable control variable.","The asymptotic alignment of BAMDP and MDP value functions suggests that in finite-sample regimes the scaling factor $\\eta_t$, rather than the bonus formula, controls the exploration–exploitation frontier; an adaptive $\\eta_t$ could replace hand-tuned schedules.","The stated rate result concerns passive data collection, whereas PTS-BE actively selects trajectories; measuring IG decay under the algorithm's own rollouts would test whether active sampling improves on the passive rate."],"forward_implications":["Information-gain bonuses are self-terminating: once the posterior is concentrated, the bonus converges to zero, so the Bayes-adaptive policy and value function return to the original MDP's.","Because the bonus targets epistemic rather than aleatoric uncertainty, it stays low in high-noise regions where the model already has data, unlike prediction-error or predictive-entropy bonuses.","In the deep-ensemble setting, EIG is computable as Jensen–Shannon disagreement, enabling look-ahead planning without per-transition posterior updates.","PTS-BE, combining trajectory sampling with EIG bonuses, is claimed to solve sparse-reward and pure-exploration tasks with substantially fewer environment steps than reactive baselines.","Under a Matérn GP prior, the bonus decays at the minimax rate $n^{-1/(2+|\\mathcal{X}|/\\alpha)}$, linking exploration cost to the smoothness and dimensionality of the dynamics."],"supporting_citations":[{"why":"Introduced the VIME information-gain intrinsic reward that this paper analyzes and provides theoretical guarantees for.","marker":"(Houthooft et al., 2016)"},{"why":"Supplies the posterior consistency theorem on which Proposition 3.1's convergence argument is built.","marker":"(Schwartz, 1965)"},{"why":"Provides the definitions and results on posterior consistency and contraction rates used in the proofs.","marker":"(Ghosal & Van der Vaart, 2017)"},{"why":"Establishes the Matérn-kernel GP posterior contraction rate used in Proposition 3.3.","marker":"(Van Der Vaart & Van Zanten, 2011)"},{"why":"Gives the minimax estimation rates that the paper identifies with the information-gain decay rate.","marker":"(Yang & Barron, 1999)"},{"why":"Introduced planning-to-explore with ensemble disagreement, the direct precursor of the EIG-as-JSD formulation and PTS-BE.","marker":"(Shyam et al., 2019)"},{"why":"Established the connection between count-based pseudo-count bonuses and Bayesian information gain, providing the comparison point for bonus decay.","marker":"(Bellemare et al., 2016)"},{"why":"Defines the prediction-error ICM baseline and motivates the noisy-TV problem that information-gain bonuses are meant to avoid.","marker":"(Pathak et al., 2017)"}],"fun_headline_variants":["Info-gain exploration bonuses fade to zero","Bayesian curiosity rewards self-terminate","Proof: exploration bonuses vanish as learning grows","Curiosity bonuses vanish exactly when they should","Exploration self-terminates via info-gain proof"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the entropy of a distribution that concentrates on a single point is taken to be zero, and that the entropy difference inherits posterior convergence without a continuity argument; differential entropy is not generally continuous under weak convergence, so this step carries the whole proof.","fun_headline_variants_meta":{"raw":{"variants":["Info-gain exploration bonuses fade to zero","Bayesian curiosity rewards self-terminate","Proof: exploration bonuses vanish as learning grows","Curiosity bonuses vanish exactly when they should","Exploration self-terminates via info-gain proof"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001228,"raw_usage":{"total_tokens":5018,"prompt_tokens":890,"completion_tokens":4128,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":4061}},"tokens_in":506,"tokens_out":4128,"duration_ms":36550,"temperature":1.0,"reasoning_tokens":4061,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:25:03.039256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a consistent posterior sequence that converges weakly to a Dirac measure but whose differential entropy does not converge to the entropy of a point mass—for example a Gaussian posterior with variance $1/n$, whose differential entropy grows like $\\tfrac12\\log(2\\pi e/n) \\to -\\infty$—and compute $H[p(\\theta|D_n)] - H[p(\\theta|D_n,s')]$ after one new observation; if this difference does not tend to zero while the posterior is consistent, the claimed implication in Proposition 3.1 fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the definitions and results on posterior consistency and contraction rates used in the proofs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduced planning-to-explore with ensemble disagreement, the direct precursor of the EIG-as-JSD formulation and PTS-BE."}],"review_version":1}