{"id":"d1a99e7e-971f-4b7d-a60d-77fa5952f95d","arxiv_id":"2607.29419","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ENTINEX identifies exploration boundaries using the entropy of a novelty-based action distribution and outperforms baseline exploration methods on sparse and delayed MuJoCo tasks.","lead":"This paper introduces ENTINEX, a reinforcement learning method that gives an agent a bonus for reaching states near the edge of the region it has already explored. The bonus is built from the entropy of an action distribution derived from state novelty, and the authors report improvements on sparse and delayed reward control tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's unquantified log-sum-exp≈max approximation is the load-bearing link; the implemented Gaussian reward may not track boundary proximity.","rationale":"The reader's weakest assumption already identifies the unquantified approximation in Eq. (4) and the possible failure of the Gaussian projection in Eq. (10). My analysis sharpens the mathematical content: Eq. (4) is precisely the statement log Z ≈ max φ', a Laplace-type approximation with no error bound, and the reverse-KL projection is mode-seeking so the implemented single-sample reward may lose even this approximate link. This is a genuine load-bearing concern because the paper's central claim is that ENTINEX rewards boundary states via entropic information; if the implemented reward is not monotonically related to boundary proximity, the reported improvements would need a different explanation. However, this concern is testable and does not by itself falsify the empirical results. Since the reader already conditioned the verdict on tightening exactly this theoretical gap and on code/verification, my concern does not change the verdict. I recommend the concrete diagnostic above as the decisive check.","tokens_in":7075,"tokens_out":6818,"duration_ms":71096,"concrete_test":"Run a controlled diagnostic in a 2D point-mass grid with a known sparse goal and known explored-region boundary: train ENTINEX (or its core modules) for a fixed number of steps, then for a grid of states s compute (i) r_int(s)=E_{aψ~ψ}[log ψ(aψ|s)] using the learned Gaussian ψ, (ii) the exact −H(ψ(·|s)) for the same ψ, (iii) the RHS of Eq. (4) using the learned dynamics/novelty, and (iv) the true distance from s to the boundary of the current empirical state-visitation support. If the Spearman correlation between (i) and (iv) is not significantly positive, or if (iii) deviates from (ii) by more than the across-state spread of (i), the central mechanism fails and the MuJoCo gains cannot be attributed to entropic boundary detection. The same check can be run on SparseWalker2d by comparing r_int with NovelD's consecutive-state novelty difference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ENTINEX's central claim rests on the assertion (Theorem 1, Eq. 4) that near the SND boundary the negative NAPD entropy equals a normalized consecutive-state novelty difference. Algebraically, the left side is E_{a~ψ}[φ'(s,a)] − log Z and the right side is E_{a~ψ}[φ'(s,a)] − max_a φ'(s,a), so Eq. (4) is the statement log Z ≈ max_a φ'(s,a) for boundary states. No error bound or 'nearness' criterion is supplied, so the theory does not establish that −H(ψ) tracks boundary proximity. The practical method then replaces the exact NAPD with the reverse-KL Gaussian projection of Eq. (10), which is mode-seeking: for a multimodal Boltzmann distribution it can lock onto one mode, and its variance (hence its entropy) can be set by the curvature of that mode rather than by the novelty gap at the boundary. The implemented reward (Eq. 8) is not even the entropy but a single-sample log-density, whose mean is −H only in expectation. Thus the load-bearing premise is that the Gaussian-projected r_int remains monotonically related to distance from the explored/unexplored boundary; this is asserted but neither proved nor measured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ENTINEX, an intrinsic-reward exploration method for sparse and delayed-reward RL. The method defines a novelty action probability distribution (NAPD) as a Boltzmann distribution over actions induced by expected consecutive-state novelty, and assigns intrinsic rewards based on the negative entropy of this distribution, intended to identify states near the boundary of the state-novelty distribution. The practical implementation replaces the NAPD with a reverse-KL Gaussian projection, uses a learned latent dynamics model to evaluate the novelty of actions not actually executed, and combines the intrinsic reward with SAC as the base algorithm. Experiments on MuJoCo environments with sparse and delayed rewards compare ENTINEX against ICM, RND, RE3, and NovelD, including a matched-novelty comparison with NovelD, an ablation over novelty functions, and an unsupervised pre-training study. The central theoretical claim is Theorem 1, which asserts an unquantified approximation relating the negative NAPD entropy to a normalized consecutive-state novelty difference near the SND boundary.","tokens_in":7445,"tokens_out":4788,"duration_ms":41523,"significance":"The core idea of using the entropy of a novelty-induced action distribution as a boundary indicator is novel and could provide a useful alternative to direct novelty-difference methods, which rely on particular action distributions. The matched-novelty comparison with NovelD is a well-designed control experiment, and the ablations on novelty functions and unsupervised pre-training broaden the empirical scope. If the empirical improvements are real, the method's practical contribution is meaningful. However, the theoretical support is currently not load-bearing: Theorem 1 is an unquantified approximation, Theorem 2 does not relate entropy to boundary proximity, and the implemented reward is a single-sample log-density whose connection to the theory is only in expectation. The experimental evidence is also reported only as learning curves without numeric tables or significance tests, making the central claim impossible to verify from the manuscript.","major_comments":[{"comment":"Theorem 1 is the load-bearing link between the proposed entropy reward and boundary proximity, but it is an unquantified approximation. Algebraically, Eq. (4) is equivalent to the statement log Z ≈ max_{a'} φ'(s,a'), with no error bound, no rate of convergence, and no formal definition of \"near the SND boundary.\" The theorem therefore does not establish that −H(ψ(·|s)) reliably tracks the consecutive-state novelty difference or the distance to the exploration boundary. Moreover, the implemented method in Section 4.2 replaces the exact NAPD with a reverse-KL Gaussian projection (Eq. 10); reverse KL is mode-seeking, so for a multimodal Boltzmann distribution the projected Gaussian can concentrate on one mode, and its entropy can be controlled by the local curvature rather than by the novelty gap at the boundary. The actual reward in Eq. (8) is a single-sample log-density whose expectation equals −H only if the projection is exact and whose per-sample variance is unspecified. The paper needs a concrete error bound for Theorem 1 and an empirical or theoretical test of the monotonic relationship between r_int and boundary proximity, for instance a correlation analysis in the sparse-reward environments.","section":"Section 4.1, Theorem 1 (Eq. 4)"},{"comment":"Theorem 2 provides a total-variation bound on the difference between expected novelty differences under two action distributions, but this bound does not involve the entropy of ψ and does not imply that entropy equals or even correlates with novelty differences. The theorem is not used in the derivation of the intrinsic reward or in any experimental analysis, so as written it does not support the paper's central claim. If the intended role of Theorem 2 is to justify replacing one action distribution with another, the bound should be connected to a quantity that actually appears in the algorithm, such as the error introduced by sampling from the Gaussian projection rather than the exact NAPD.","section":"Section 4.1, Theorem 2 (Eq. 5)"},{"comment":"The policy update uses the tuple {(s_i, a_r, s'_i, r_total_i)}, but the variable a_r is undefined anywhere in the algorithm or the surrounding text. It appears to be a typo for a_i, the action stored in the replay buffer, but as written the algorithm is not reproducible. Please correct the variable and specify whether the policy update uses the executed action a_i or some other action.","section":"Section 4.2, Algorithm 1, line 14"},{"comment":"The central empirical claim of \"consistent improvement\" over baselines is supported only by learning-curve plots. The manuscript does not provide numerical tables of mean and standard deviation at the final training step or at any intermediate points, and no significance tests or effect sizes are reported. The caption of Fig. 3 mentions 5 random seeds, but the shaded area only shows 0.5σ, which underrepresents the variance. For example, the claim that baselines \"fail to learn effectively\" in SparseWalker2d is not backed by final scores or error bars, so the reader cannot assess whether the differences are reliable. Please add tables with mean ± std over all seeds for every method and environment, and ideally report the number of seeds and a statistical comparison or confidence intervals.","section":"Section 5.2, Figures 3–6"},{"comment":"The matched-novelty comparison with NovelD is intended to isolate the effect of the boundary-estimation method, but the description does not specify the exact hyperparameters, network sizes, or the underlying novelty estimator used for both methods in this comparison. Since the default novelty functions are different (RE3 for ENTINEX and RND for NovelD), the claim that any performance gap primarily arises from the entropic approach requires that all other components be tightly controlled. Please provide the precise experimental configuration for Fig. 4 and discuss any remaining differences that could affect the comparison.","section":"Section 5.2, Figure 4"}],"minor_comments":[{"comment":"The symbol ψ is used for both the exact Boltzmann NAPD in Eq. (3) and the parameterized Gaussian projection in Eq. (10), which is confusing. Using a different symbol for the projected distribution (e.g., ψ_param) would make the derivation of Eq. (11) clearer.","section":"Section 2.2, Eq. (3) and Section 4.2, Eq. (10)"},{"comment":"In the stated loss, the same notation ψ appears inside the expectation and inside the log-density; after the projection, the sampling distribution and the evaluated density should be the same parameterized Gaussian, but the current notation obscures this. Please rewrite the loss using explicit parameterization.","section":"Section 4.2, Eq. (11)"},{"comment":"The term \"state-novelty distribution (SND)\" is used throughout but never formally defined; the definition of its boundary is also informal. A precise definition would make the premise of Theorem 1 testable and would help readers understand exactly what the boundary is.","section":"Section 4.1, State-Novelty Distribution"},{"comment":"The discussion mentions \"training instability and additional computational overhead\" but does not quantify the overhead of the dynamics model and the additional components relative to the baselines. A brief cost comparison (wall-clock time or parameter count) would strengthen the practical assessment.","section":"Section 6, Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper presents an interesting and potentially valuable exploration method, and the matched-novelty comparison is a thoughtful experimental design. However, the theoretical justification is currently too weak to support the central claim, the algorithm contains an undefined variable, and the experimental evidence is not reported in a sufficiently verifiable form. These issues are fixable within the scope of a revision, so I recommend major revision rather than rejection. In particular, I would require an error-bound or a rigorous reformulation of Theorem 1, a correction of Algorithm 1, and the addition of numeric tables with error bars and statistical comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea is genuinely new: instead of measuring boundary proximity through novelty differences between consecutive states, they build a Boltzmann distribution over actions weighted by expected next-state novelty and use its entropy as the exploration signal. That is a real departure from NovelD, BeBold, and RIDE. Second, the theoretical bridge in Theorem 1 is thinner than it looks, and the experimental appendix is too sparse to bank on. That combination, not the idea itself, is what needs scrutiny.\n\nWhat the paper does well: the matched-novelty comparison against NovelD in Figure 4 is the right move. It isolates the boundary-detection mechanism from the choice of novelty estimator, which is exactly where prior comparisons get muddy. The unsupervised pre-training experiment is a reasonable bonus, and the Discussion is honest that rigorous theoretical characterization remains open.\n\nNow the soft spots, in order of severity.\n\nTheoretical: Theorem 1 states that near the boundary the negative NAPD entropy equals a normalized consecutive-state novelty difference. Algebraically that is the claim log Z ≈ max_a φ'(s,a), with no error bound and no nearness criterion. So the theory does not establish that −H(ψ) tracks boundary proximity; it is a heuristic dressed as a theorem. Worse, the practical method replaces the exact Boltzmann NAPD with a reverse-KL Gaussian projection (Eq. 10), which is mode-seeking: for a multimodal NAPD it can lock onto one mode, and its entropy then reflects local curvature, not the novelty gap. And the implemented reward (Eq. 8) is a single sample of log ψ, whose mean is −H only in expectation. The load-bearing premise—that the Gaussian-projected reward remains monotonically related to boundary distance—is asserted, not proved or measured.\n\nExperimental: all results are learning curves with 5 seeds, no numeric tables, no significance tests, and no code release. One number (SparseWalker2d final returns) is given, but I cannot verify the \"consistently outperforms\" claim from shaded curves alone. There is also an undefined variable in Algorithm 1: the update buffer is written with a_r while the sampling loop uses a_i. Likely a typo, but it signals the manuscript was not cleaned.\n\nOn the circularity worry: the reader's stress test correctly notes that NAPD entropy is a deterministic function of novelty and reduces to NovelD's signal under the Theorem 1 approximation. That is fair. But the implemented method—learned Gaussian projection, dynamics model, sample-based log-density—does behave differently experimentally, and the matched-novelty comparison is exactly the right way to claim a mechanism. So I do not see it as fatal; I see it as an unresolved need to characterize when the entropic signal actually diverges from the novelty-difference signal.\n\nVerdict: this paper deserves a serious referee. The idea is worth engaging, and the experiments are plausible but under-reported. A referee should require code and hyperparameters, numeric tables with variance, and a rewriting of Theorem 1 as a conjecture or with a proven bound. I would send it to peer review, expecting major revision.","headline":"Entropic boundary exploration is a genuinely new idea with a good matched-novelty comparison, but Theorem 1 is an unquantified approximation and the empirical reporting is too thin to fully trust.","tokens_in":7812,"tokens_out":3069,"would_cite":false,"duration_ms":26473,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Entropy-based intrinsic rewards push RL agents past explored boundaries","keywords":["reinforcement learning","exploration","intrinsic reward","entropy","state novelty","boundary-based exploration","sparse rewards","delayed rewards"],"falsifier":"In a small tabular MDP with known transition probabilities, compute both the negative entropy of the NAPD exactly and the quantity on the right-hand side of Theorem 1 for every state; if the two diverge for states that are not 'near the boundary', or if replacing the entropy reward with the exact normalized novelty difference achieves the same performance, then the entropic computation is not the operative cause.","tokens_in":6857,"feed_emoji":"🧭","tokens_out":5086,"duration_ms":41325,"temperature":0.7,"pith_summary":"ENTINEX is a reinforcement-learning exploration method that identifies states near the boundary between explored and unexplored territory and gives the agent an intrinsic reward for being there. Instead of comparing novelty between consecutive states, as earlier boundary methods do, it computes the entropy of a Boltzmann distribution over actions that favor novel transitions: near the boundary, actions split into those that stay inside and those that lead out, making the distribution sharply peaked and its negative entropy large. The paper reports that this entropic signal consistently outperforms established exploration baselines in MuJoCo tasks with sparse or delayed rewards, and that the improvement persists even when the novelty function is matched with the baseline. The broader value is a parameter-light way to push agents into genuinely new regions without reward shaping or dense feedback.","feed_headline":"Entropy of action distribution guides agents past explored limits","feed_subtitle":"New method ENTINEX spurs RL agents to cross state-novelty boundaries, beating ICM, RND, RE3, and NovelD.","key_machinery":"The novelty action probability distribution (NAPD): a Boltzmann distribution over actions, $\\psi(a|s) \\propto \\exp(\\phi'(s,a))$, where $\\phi'$ is the expected novelty of the next state. Its sample-based negative log-probability $r_{\\mathrm{int}}(s) = \\log \\psi(a_\\psi|s)$ is the intrinsic reward. The NAPD converts the scalar novelty landscape into an action-space distribution whose shape (peaked vs. uniform) marks the boundary; Theorem 1 connects its negative entropy to the normalized consecutive-state novelty difference, while Theorem 2 bounds the discrepancy from sampling actions from a different distribution. For continuous action spaces, the Boltzmann distribution is projected onto a parameterized Gaussian via KL minimization (Eq. 10), and a learned latent dynamics model supplies $\\phi'(s,a_\\psi)$ for actions the agent never executed.","core_discovery":"The central claim is that the entropy of the novelty action probability distribution (NAPD) — the Boltzmann distribution over actions that favors actions leading to novel states — encodes whether a state lies on the state-novelty boundary, and that using its negative log-probability as an intrinsic reward steers a policy to cross that boundary. Theorem 1 states that near the boundary the negative entropy of the NAPD is approximately the normalized consecutive-state novelty difference, the same quantity earlier boundary-based methods compute directly from visited transitions. ENTINEX separates the roles: the NAPD tells you which actions would be novel, and the entropy tells you whether that novelty is concentrated or spread out. The paper reports consistent wins over ICM, RND, RE3, and NovelD across sparse-reward and delayed-reward MuJoCo environments, and shows that the entropic boundary estimate, rather than the novelty measure, drives the improvement against NovelD.","pith_inferences":["If Theorem 1 could be strengthened with an explicit error bound (the paper gives none), the NAPD entropy would become a provable proxy for boundary distance, potentially enabling convergence guarantees for exploration bonuses.","The method's dependence on a learned dynamics model to score never-taken actions suggests a natural stress test: in environments where the latent dynamics model is inaccurate, the intrinsic reward may mis-rank actions and exploration could degrade—an untested edge case.","The entropy-of-induced-distribution idea is portable beyond RL: the same NAPD construction could serve as an acquisition function in active learning or as a curiosity signal in unsupervised skill discovery, wherever one wants to detect decision boundaries from a novelty field.","A simpler variant that replaces the Gaussian projection with a direct entropy estimate (e.g., via sampling) could test whether the projection step, rather than the entropy itself, carries the empirical gains."],"forward_implications":["Agents trained with ENTINEX's entropy-based intrinsic reward learn faster and reach higher returns than prediction-error (ICM, RND), state-entropy (RE3), and consecutive-novelty (NovelD) baselines in sparse- and delayed-reward continuous-control tasks.","The boundary signal comes from the entropy of the action distribution, not from the specific novelty estimator, so ENTINEX works with both random-encoder entropic novelty and prediction-error novelty.","Because the intrinsic reward uses actions sampled from the NAPD rather than from the behavior policy, it guides exploration toward novel transitions without requiring the agent to actually take those actions.","Unsupervised pre-training with only ENTINEX's intrinsic rewards creates a policy that, when fine-tuned on task rewards, learns with fewer environment interactions.","When the novelty function is fixed, ENTINEX's entropic boundary estimate still beats NovelD's consecutive-state novelty difference, isolating the boundary-identification mechanism as the source of improvement."],"supporting_citations":[{"why":"Provides the default state-novelty function (the random encoder of RE3) and serves as the state-entropy baseline.","marker":"[27]"},{"why":"Provides the prediction-error novelty estimator used as an alternative novelty function and in the NovelD baseline.","marker":"[3]"},{"why":"The consecutive-state novelty-difference boundary baseline used for the matched-novelty comparison.","marker":"[32]"},{"why":"ICM prediction-error baseline against which ENTINEX is compared.","marker":"[24]"},{"why":"The latent dynamics model used to predict next states for actions sampled from the NAPD.","marker":"[12]"},{"why":"Supplies the concurrent training strategy for the dynamics model.","marker":"[8]"}],"fun_headline_variants":["Entropy of novel-action distribution guides RL past state boundaries","ENTINEX: entropy of novel action choices finds state exploration edges","Entropic boundary reward beats ICM, RND, RE3 in delayed-reward tasks","Entropy of novel-action distribution pushes agents across state novelty boundaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Near a state-novelty boundary the novelty action probability distribution is sharply peaked enough that its negative entropy tracks the normalized novelty difference between consecutive states, and the Gaussian projection of Eq. (10) preserves that peak.","fun_headline_variants_meta":{"raw":{"variants":["Entropy of novel-action distribution guides RL past state boundaries","ENTINEX: entropy of novel action choices finds state exploration edges","Entropic boundary reward beats ICM, RND, RE3 in delayed-reward tasks","Entropy of novel-action distribution pushes agents across state novelty boundaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000935,"raw_usage":{"total_tokens":3951,"prompt_tokens":849,"completion_tokens":3102,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":3026}},"tokens_in":465,"tokens_out":3102,"duration_ms":20838,"temperature":1.0,"reasoning_tokens":3026,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:22:04.808095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a small tabular MDP with known transition probabilities, compute both the negative entropy of the NAPD exactly and the quantity on the right-hand side of Theorem 1 for every state; if the two diverge for states that are not 'near the boundary', or if replacing the entropy reward with the exact normalized novelty difference achieves the same performance, then the entropic computation is not the operative cause.","supporting_citations":[{"cited_title":"In: International Conference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Provides the default state-novelty function (the random encoder of RE3) and serves as the state-entropy baseline."},{"cited_title":"Advances in Neural Information Processing Systems34, 25217–25230 (2021)","cited_arxiv_id":null,"evidence_quote":"The consecutive-state novelty-difference boundary baseline used for the matched-novelty comparison."},{"cited_title":"In: International Conference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the concurrent training strategy for the dynamics model."}],"review_version":1}