{"id":"882096b3-ad3b-46b9-9086-243f28118cad","arxiv_id":"2411.14085","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RAMP maximizes the divergence between an agent's current and past state distributions, a proxy for Shannon entropy, and outperforms several prior exploration methods on continuous-control benchmarks.","lead":"This paper introduces RAMP, a reinforcement-learning exploration method that rewards an agent for moving away from states it visited before. RAMP turns the goal of maximizing state diversity into a simple classification or distance-maximization bonus and shows strong coverage in maze, robot locomotion, and manipulation benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim hinges on the policy-entropy proxy for occupancy entropy, but the successful RAMPW variant is not covered by Theorem 1, and the KL-based RAMPKL collapses in high-dimensional control; a direct occupancy-entropy measurement would decide whether RAMP truly maximizes state entropy.","rationale":"The paper has real strengths: Theorem 1 is correctly derived (Appendix B), the algorithm is clearly specified, and RAMPW's coverage results on Ant and Humanoid are striking. But the central claim in the abstract—that exploration is cast as maximizing Shannon entropy of the state occupation measure—is only formally connected to the KL objective through Theorem 1, and that connection requires the policy-entropy proxy. The paper admits the proxy is not universal, and its own Proposition 4 demonstrates a mode-collapse failure for the KL objective. The empirical evidence reinforces this: RAMPKL, the variant with the theoretical backing, fails on exactly the high-dimensional locomotion tasks where RAMPW excels. RAMPW, the successful variant, is justified only heuristically and its objective is not a lower bound on entropy increase. This means the theoretical narrative and the empirical star are decoupled. A direct measurement of occupancy entropy would settle whether the method actually does what the title claims. This is a substantial but addressable issue, consistent with the reader's CONDITIONAL verdict; I do not see grounds to reject, as the method may still be a useful exploration heuristic with strong results on some tasks, but the paper should either establish the proxy or reframe the claims away from entropy maximization.","tokens_in":19670,"tokens_out":12033,"duration_ms":120525,"concrete_test":"Train RAMPW, RAMPKL, APT, and SAC on Ant and HalfCheetah with the paper's hyperparameters; at convergence, estimate the differential Shannon entropy of the state occupancy distribution from the accumulated replay buffer using a k-NN estimator (e.g., Kozachenko-Leonenko) on raw states, and also on torso-coordinate projections. If RAMPW's raw-state occupancy entropy is not significantly higher than SAC's despite its xyz coverage advantage, the claim that RAMP maximizes Shannon entropy of the state occupation measure fails, and the coverage gains should be attributed to the temporal-distance objective's alignment with the evaluation metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The entire theoretical edifice rests on Theorem 1's lower bound, which requires H_ρπ[S] in the objective. The paper replaces it with the policy entropy E_{a∼π}[−log π(a|s)] and calls this a 'classic proxy,' explicitly admitting it may not hold universally. This is not a minor technicality: Proposition 4 (Appendix I) shows that without a term enforcing state coverage, the KL repulsive term is maximized by distributions supported on zero- or low-density regions of the past mixture µn, which can be arbitrarily low-entropy point masses. The policy-entropy bonus is therefore the only mechanism preventing collapse, and its efficacy is environment-dependent. The paper's own Table 1 shows this failure mode: RAMPKL achieves only 1.2% relative coverage on Ant (vs 78.35% for RAMPW) and 29.76% on HalfCheetah. Meanwhile, the version that performs well, RAMPW, is explicitly not a lower bound on the entropy increment (Section 2: 'this objective does not maximize a lower bound on ∆n per se'). Thus the empirical success of RAMP does not validate the Shannon-entropy-maximization claim; it validates a temporal-distance heuristic. The coverage metric used in Table 1 (xyz torso coordinates) is also the quantity that the temporal-distance Wasserstein reward most directly encourages, so the headline results may reflect metric alignment rather than general state-occupancy entropy increase.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAMP, an exploration method for RL that maximizes a divergence between the agent's current state-occupancy measure and a discounted mixture of past occupancies, plus a policy-entropy bonus. Two instantiations are presented: RAMPKL, which uses a contrastive classifier to estimate a log-density ratio, and RAMPW, which uses a Wasserstein critic with a temporal-distance Lipschitz constraint. The authors prove a lower bound on the increment of the Shannon entropy of the past-mixture distribution, provide two monotonic-improvement theorems for the approximate reward models, and evaluate coverage and episodic returns on maze, MuJoCo locomotion, and Fetch tasks.","tokens_in":19945,"tokens_out":9495,"duration_ms":89362,"significance":"If the central claim held, RAMP would offer a simple, classifier- or critic-based way to increase state-occupancy entropy without expensive density estimation, and the empirical comparison is useful, especially RAMPW's strong coverage on Ant. The paper is also honest about identifying where KL-based exploration collapses. However, the theoretical bridge from divergence maximization to Shannon-entropy maximization is incomplete for both variants: the variant that works well, RAMPW, is explicitly not covered by Theorem 1, and the variant that is covered, RAMPKL, fails on high-dimensional locomotion. The paper therefore does not currently substantiate its abstract-level claim that it maximizes the Shannon entropy of the state occupation measure.","major_comments":[{"comment":"Theorem 1's lower bound contains Hρπ[S], the Shannon entropy of the state-occupancy measure, but the proposed objective in Eq. (2) replaces this term with the policy-entropy bonus λA Es∼ρπ,a∼π[−log π(a|s)]. The paper states that this proxy 'may not universally apply across all environments,' and Appendix I's Proposition 4 shows that the KL repulsive term alone is maximized by concentrating mass where the past mixture µn has zero density, which can reduce rather than increase occupancy entropy. Table 1 shows the concrete failure mode: RAMPKL obtains only 1.2% relative coverage on Ant and 29.76% on HalfCheetah. The central claim that RAMP maximizes Shannon state entropy is therefore not supported without a direct measurement of Hρπ[S] over training; I recommend adding such a measurement and comparing with a policy-entropy-only baseline.","section":"Section 2, Eq. (2), and Theorem 1"},{"comment":"The version that actually performs well, RAMPW, is explicitly not maximizing a lower bound on the entropy increment: Section 2 states 'this objective does not maximize a lower bound on ∆n per se.' Its strong coverage results (78.35% on Ant, 74.43% on Hopper) may reflect alignment between the temporal-distance Wasserstein reward and the xyz-coordinate coverage metric used in Table 1, rather than an increase in the Shannon entropy of the state-occupancy measure. To separate these explanations, the paper should report coverage on state features not directly predicted by the temporal-distance potential (e.g., joint configurations) and measure Hρπ[S] directly.","section":"Section 2, Eq. (4), and Section 5.2, Table 1"},{"comment":"The statement and proof of Theorem 2 are inconsistent. The theorem defines ε0 by ∥ρπ′/ρπ − 1∥∞ ≥ ε0 and calls ρπ′ 'close to ρπ', but the proof immediately assumes 1−ε0 ≤ ρπ′/ρπ ≤ 1+ε0, which requires the norm to be at most ε0. As written, the premise does not express closeness, and the conclusion does not follow. The proof also contains an unjustified step: from |ˆrDKL(s)−f(ρ,µn)(s)|≤ε1 it does not follow that ⟨ρ,f(ρ,µn)⟩≥−ε1. The theorem and proof need to be corrected before this result can serve as a guarantee for RAMPKL's reward model.","section":"Appendix C, Theorem 2"},{"comment":"The derivation of the estimated reward fϕ relies on equal prior probabilities P(L=1)=P(L=0). The algorithm and appendices never specify how the positive and negative samples are balanced; in Algorithm 1, Dρ and Dµn are updated in different ways (episode-based collection versus accept-reject), so the class priors are not equal in general. If the batches are not explicitly balanced, the learned fϕ is not the log-density ratio in Eq. (2), and Theorem 2's approximation-error assumption is not satisfied. Please state the exact sampling procedure used in the experiments.","section":"Section 3.2, Eq. (6), and Appendix G"}],"minor_comments":[{"comment":"The aggregate mean scores are reported as '33, 527×10^3 and 32, 408×10^3', but summing the five entries in Table 2 gives 33,527 and 32,408; the ×10^3 factor appears erroneous and should be corrected.","section":"Section 5.3, paragraph after Table 2"},{"comment":"The notation W(ρπ, βρπ + µn(β − 1)) is confusing; it should be written as βρπ + (1 − β)µn to match Eq. (3), since µn(β − 1) is not defined as a distribution.","section":"Appendix D and Theorem 3"},{"comment":"The term '1-Lipshitz' should be '1-Lipschitz'.","section":"Equation (3)"},{"comment":"Eq. (6) defines LDKL as a loss to be minimized, but Algorithm 1 writes 'ϕ = arg max_ϕ LDKL(ϕ)'; please clarify whether the displayed quantity is the negative loss or the classification objective in Appendix H, and align the notation.","section":"Algorithm 1"},{"comment":"The update of Dµn uses a Bernoulli accept-reject procedure, but the paper does not state the buffer capacity or the exact number of samples Neρ used; these details are needed for reproducibility.","section":"Section 3.4 and Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the empirical study is substantial. My main concern is that the theoretical claims in the abstract and Section 2 are stronger than what the paper actually establishes: RAMPW is best described as a temporal-distance exploration heuristic rather than a Shannon-entropy maximizer, and the theorem that supports RAMPKL has a faulty premise. The experiments are promising enough to justify a revision rather than rejection, but the authors should either add a direct measurement of occupancy entropy or substantially soften the entropy-maximization claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RAMP is a clean, well-written exploration method. The genuinely new bit is the iterative objective: maximize the KL or temporal Wasserstein divergence between the current occupancy measure and a discounted mixture of past occupancies, estimated by a simple classifier or a Wasserstein critic. That specific formulation is not in the cited prior work. Theorem 1 is correctly derived and the contrastive-learning reduction for the KL reward is clearly explained. The maze results and the strong RAMPW numbers on Ant and Humanoid are also credible; the Ant coverage gain over METRA is striking.\n\nThe soft spots are real but not fatal. The central claim that this maximizes Shannon entropy of the state occupation measure rests on replacing H_rho[S] in Theorem 1 with policy entropy, a proxy the authors admit may not hold universally. RAMPKL collapsing on Ant to 1.2% relative coverage is exactly that failure mode. The stress-test note is right: the successful variant, RAMPW, is not a lower bound on the entropy increment at all, and the coverage metric (xyz torso coordinates) is closely aligned with what the temporal-distance Wasserstein reward directly encourages. So I'd treat RAMPW as a well-motivated temporal-distance heuristic with good empirical coverage, not as a validated entropy maximizer. Proposition 4 also shows the KL objective can concentrate mass on zero-density regions of the past mixture, which need not be high-entropy. The paper honestly flags some of this, which I appreciate.\n\nSmaller issues: the promised GitHub repository has no URL, the hyperparameter detail is thin, and I would like a direct measurement of occupancy entropy (or a featurized version) to decide whether the Shannon-entropy story adds anything beyond \"temporal distance gives good coverage.\" These are addressable in revision, not reasons to dismiss the work.\n\nWho should read it: people working on intrinsic motivation, state coverage, and unsupervised skill discovery. It deserves a serious referee. I'd send it out and ask for (a) an occupancy-entropy measurement or a featurized-state comparison, (b) explicit scope for RAMPW relative to Theorem 1, and (c) code and full hyperparameters. The paper is useful even if the theoretical framing is oversold.","headline":"A useful exploration heuristic with a solid classifier-based formulation; the entropy-maximization interpretation is only partially supported, but the empirical case for RAMPW is real and worth a serious look.","tokens_in":20495,"tokens_out":1576,"would_cite":true,"duration_ms":18622,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that maximizing the divergence between the agent's current state distribution and a discounted mixture of its past distributions, plus a policy-entropy bonus, is an effective proxy for maximizing the Shannon entropy of…","keywords":["reinforcement learning","intrinsic motivation","exploration","Shannon entropy","state occupancy measure","KL divergence","Wasserstein distance","temporal distance"],"falsifier":"Run RAMPKL in a continuous environment where every random action keeps the agent near its start position but a single deterministic action reaches a distant never-visited state. If the intrinsic reward increases while discretized state-space coverage stays flat or falls, the policy-entropy proxy rather than true state diversity is carrying the objective.","tokens_in":19448,"feed_emoji":"🤖","tokens_out":9764,"duration_ms":82353,"temperature":0.7,"pith_summary":"This paper tries to establish that an agent can explore by actively distancing itself from its own past, rather than by directly estimating the entropy of its state distribution. The proposed RAMP method rewards states that separate the current occupancy measure $\\rho_{n+1}$ from the discounted past mixture $\\mu_{n+1}=\\beta\\rho_{n+1}+(1-\\beta)\\mu_n$, together with a policy-entropy bonus. A theorem lower-bounds the per-epoch entropy increase by this divergence, and the divergence itself is estimated either by a contrastive classifier (RAMPKL) or by a temporal-distance Wasserstein critic (RAMPW). If the claim holds, exploration in continuous control reduces to training one classifier or critic against a replay buffer, and the paper's experiments report state coverage competitive with or better than several existing exploration methods on mazes, locomotion, and manipulation tasks.","feed_headline":"Running from past states wins exploration benchmarks","feed_subtitle":"It turns state coverage into a classifier or transport problem and beats prior exploration on mazes and locomotion.","key_machinery":"The load-bearing object is the discounted past mixture $\\mu_{n+1}(s)=\\beta\\rho_{n+1}(s)+(1-\\beta)\\mu_n(s)$, which compresses all previous behavior into one distribution that a replay buffer can sample. Theorem 1 shows the entropy difference $H_{n+1}-H_n$ is bounded below by $\\beta(D_{\\mathrm{KL}}(\\rho_{n+1}\\|\\mu_{n+1}) + H_{\\rho_{n+1}}[S] - H_n)$, turning entropy maximization into a divergence-maximization problem. For RAMPKL, the density ratio inside the KL term is estimated by a contrastive classifier whose pre-sigmoid output is the state reward. For RAMPW, the repulsion is quantified by a Wasserstein distance whose dual potential $f_\\phi$ is trained under a 1-Lipschitz constraint with respect to the temporal distance, making the reward grow with the minimum number of environment steps separating a current state from past states.","core_discovery":"The central discovery is that the Shannon entropy of the agent's experience distribution can be increased monotonically by repeatedly maximizing a divergence between present and past, and this divergence can be learned with a single network. Concretely, Theorem 1 gives $H_{n+1}-H_n \\ge \\beta\\big(D_{\\mathrm{KL}}(\\rho_{n+1}\\|\\mu_{n+1}) + H_{\\rho_{n+1}}[S] - H_n\\big)$, so the authors set the next policy to maximize the repulsive term with an entropy regularization. RAMPKL learns the density ratio inside the KL through binary classification between current and past states, while RAMPW maximizes the Wasserstein distance under a 1-Lipschitz constraint in the temporal distance. In the paper's experiments, RAMPW reaches the highest discretized state-space coverage on four of five locomotion tasks, including a score on Ant six times that of the second-best baseline, whereas RAMPKL remains competitive when an extrinsic reward guides exploration.","pith_inferences":["Editorial inference: the KL variant's failure mode is the one Proposition 4 predicts—optimal solutions place mass where the past mixture $\\mu_n$ has zero mass—so RAMPKL should be expected to underperform whenever a novel joint configuration is easier to reach than a novel location.","Editorial inference: the comparison between RAMPKL and RAMPW hints that exploration gains decompose into a metric component and a divergence component; replacing the temporal distance with a Euclidean or task-specific latent metric would test which component drives each benchmark's improvement.","Editorial inference: because the objective compares occupancy measures rather than trajectories, it does not by itself reward diverse behaviors that visit identical states; conditioning the same repulsion on skill descriptors would be a natural extension for skill discovery."],"forward_implications":["If Theorem 1's proxy holds, any policy optimization method can add exploration by training a single classifier or critic against a replay buffer of past states, bypassing full-state density estimation.","On the paper's experiments, RAMPW achieves the highest state-space coverage on four of five locomotion environments, with an Ant score about six times the second-best baseline.","When an extrinsic reward is present, the paper reports that RAMPKL's aggregate score across five locomotion environments is slightly higher than RAMPW's, while RAMPW exceeds the second-best baseline on Ant by more than 40 percent.","In high-dimensional locomotion, the KL objective can be satisfied by changing joint configurations without translating the agent; the Wasserstein version with temporal distance is the authors' fix for this failure mode.","The authors suggest the same running-away-from-the-past objective can be combined with skill-based or hierarchical exploration strategies."],"supporting_citations":[{"why":"Supplies the contrastive classification view that turns the log-density ratio in the KL objective into a learned reward.","marker":"Eysenbach et al. (2020)"},{"why":"Supplies the adversarial intrinsic motivation and temporal-distance Lipschitz constraint used to train the Wasserstein critic in RAMPW.","marker":"Durugkar et al. (2021)"},{"why":"Provides METRA, the temporal-distance Wasserstein method that RAMPW builds on and is compared against.","marker":"Park et al. (2023b)"},{"why":"Provides APT, the main entropy-maximizing baseline and the density-estimator approach RAMP avoids.","marker":"Liu & Abbeel (2021)"},{"why":"Supplies Soft Actor-Critic, the policy optimizer used to maximize the intrinsic rewards for RAMP and the baselines.","marker":"Haarnoja et al. (2018)"},{"why":"Supplies the prior entropy-maximization result that RAMP reformulates into a divergence objective.","marker":"Hazan et al. (2019)"},{"why":"Provides the Kantorovich duality used to define the Wasserstein distance in the RAMPW objective.","marker":"Villani et al. (2009)"},{"why":"Provides LSD, a Lipschitz-constrained skill-discovery baseline compared in the locomotion and maze experiments.","marker":"Park et al. (2022)"}],"fun_headline_variants":["Run from past states to explore new ones","RAMP turns exploration into a repulsion game","Wasserstein distance trumps KL for state coverage","Maximize entropy by fleeing your history","Explore by outrunning your own footprints"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that rewarding the agent for high-action randomness (policy entropy) actually spreads the states it visits; if random actions do not widen the state distribution, the 'run from the past' reward can be satisfied by revisiting a few states that were never seen before, and coverage can fall instead of rise.","fun_headline_variants_meta":{"raw":{"variants":["Run from past states to explore new ones","RAMP turns exploration into a repulsion game","Wasserstein distance trumps KL for state coverage","Maximize entropy by fleeing your history","Explore by outrunning your own footprints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1537,"prompt_tokens":923,"completion_tokens":614,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":547}},"tokens_in":539,"tokens_out":614,"duration_ms":6503,"temperature":1.0,"reasoning_tokens":547,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:33:22.565040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RAMPKL in a continuous environment where every random action keeps the agent near its start position but a single deterministic action reaches a distant never-visited state. If the intrinsic reward increases while discretized state-space coverage stays flat or falls, the policy-entropy proxy rather than true state diversity is carrying the objective.","supporting_citations":[],"review_version":1}