{"id":"5a6b443d-d84b-4773-9f95-121772687d0d","arxiv_id":"2506.05615","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Maximum entropy RL can be formally steered into arbitrary suboptimal policies at convergence by adding entropy trap states, while standard RL is unaffected.","lead":"Maximum entropy training can steer a reinforcement learning agent toward random, low-reward behavior in control tasks that demand precise actions. This paper builds a formal entropy trap and uses it to explain why SAC fails on realistic vehicle, quadrotor, and quadruped benchmarks where PPO succeeds.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The theorem's proof is unsound: Lemma B.1 constructs Q(s,a)=α log π(a|s)+v_s without checking realizability, but in Definition 5.1 all actions mapping to the same successor s′ share one state s_μ, forcing Q(s,a) to be constant over that class.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing gap I would flag, and it is fatal for Theorem 5.5. Definition 5.1 reroutes every action leading to a successor s′ through one new state s_μ=μ(s′). Therefore the Q-function at s after extension is constrained to Q(s,a)=r_M(s,a)+γV(s_μ), which is indexed only by the successor class, not by the individual action. Lemma B.1 instead directly assigns Q(s,a)=α log π(a|s)+v_s, which can differ between actions in the same successor class, and the proof never checks that this assignment satisfies the Bellman equation of the extended MDP. The minimal two-action, single-successor MDP is an explicit counterexample: the induced MaxEnt policy is uniform, so an arbitrary target π cannot be achieved. Theorem B.3 compounds this by writing V(s_μ)=Q(s,a)/P(s′|s,a), which is not a valid way to define V(s_μ) when several actions map to the same successor and is undefined when the transition probability is zero. I therefore agree with the REJECT verdict. This is not an attack on the empirical portion: the toy example and experiments can plausibly support a weaker qualitative claim that entropy can bias policy optimization toward low-return policies in environments with precise, low-entropy optimal regions. But the paper's stated formal main result, that an arbitrary policy can be made MaxEnt-optimal at a targeted state without changing optimal policies elsewhere, is not sound as written.","tokens_in":21899,"tokens_out":5190,"duration_ms":59671,"concrete_test":"Analytical counterexample: instantiate the minimal MDP S={s,s_T}, A={a1,a2}, P(s_T|s,a_i)=1, r≡0, γ=0.99. Apply the Entropy Bifurcation Extension of Definition 5.1 at s with target π(a1)=0.9, π(a2)=0.1. Write the Bellman fixed-point equation at s: Q(s,a_i)=r(s,a_i)+γV(s_μ)=γV(s_μ) for both actions, so induced MaxEnt policy is uniform over {a1,a2}. Since V(s_μ) is the only tunable parameter entering Q(s,·) and enters identically for both actions, no choice of extension parameters can produce the target. This falsifies Theorem 5.5. To rule out the reward caveat, repeat with nonzero r(s,a1)≠r(s,a2); the induced policy is then forced to match the fixed reward difference, still not arbitrary.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing concern is a realizability gap in the proof of Theorem 5.5. Lemma B.1 proves backward compatibility by setting Q(s,a)=α log π(a|s)+v_s (Equation 5), which makes the target policy a fixed point of the softmax update. However, this Q-function is not shown to be realizable by the extension defined in Definition 5.1. In that extension, every action a with P(s′|s,a)>0 is rerouted through a single new state s_μ=μ(s′), and rewards on original states are unchanged. Hence the actual Bellman equation at s is Q(s,a)=r_M(s,a)+γV(s_μ): the Q-value depends on the successor s′ through s_μ, not on the individual action beyond the fixed reward r_M(s,a). When r_M(s,a) is constant over actions in the same successor class, Q(s,a) is exactly constant on that class, so the induced MaxEnt-optimal policy is uniform within the class. Thus an arbitrary target distribution π(·|s) cannot be induced unless it happens to be constant on every successor class. Concretely, take M with S={s,s_T}, A={a1,a2}, P(s_T|s,a_i)=1, and r≡0. Definition 5.1 adds one state s_μ; Q(s,a1)=γV(s_μ)=Q(s,a2), so the MaxEnt policy at s is (1/2,1/2), and no choice of |A_1^μ|, |A_2^μ|, or r(s_T^μ) can achieve π=(0.9,0.1). Theorem 5.5 therefore fails as stated. The proof of Theorem B.3 compounds this by writing V(s_μ)=Q(s,a)/P(s′|s,a), which is not a valid definition when multiple actions share the same successor and is undefined for transitions with probability zero.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that maximum-entropy RL can be systematically misled in control problems that require low-entropy policies. It presents a toy MDP, a general construction called the Entropy Bifurcation Extension, and a theorem (Theorem 5.5) claiming that for any MDP the extension can make an arbitrary policy at a target state the MaxEnt-optimal policy while leaving all other states' optimal policies unchanged. It then reports qualitative experiments comparing SAC and PPO on vehicle, quadrotor, quadruped, and benchmark tasks, plus an adaptive-entropy variant (SAC-AdaEnt). The toy calculation in Appendix A.1 is analytically correct, but the proof of the general theorem contains a load-bearing realizability gap: the Q-function constructed in Lemma B.1 is not shown to be realizable under the transition structure of Definition 5.1, and in the simplest MDPs it cannot be.","tokens_in":22348,"tokens_out":11302,"duration_ms":113182,"significance":"If Theorem 5.5 were true, it would be an important formal caveat: MaxEnt methods could converge to arbitrarily bad policies while standard RL remains unaffected, with direct implications for SAC and related algorithms. The paper also contains a useful pedagogical toy example and plausible qualitative evidence that entropy regularization can hurt in tasks with narrow feasible action regions. However, the central theoretical claim is unsound as stated. The toy example and experiments do not repair the gap, because the experiments are not instances of the formal construction and the failure of the theorem is structural rather than a matter of proof detail.","major_comments":[{"comment":"The proof of Lemma B.1 constructs Q(s,a)=α log π(a|s)+v_s (Eq. (5) in the appendix) and declares the target policy MaxEnt-optimal, but it never verifies that this Q is a Bellman backup in the extended MDP. Under Definition 5.1, every action with positive transition to a given successor s' is rerouted to the same new state s_μ=μ(s'), so the actual backup at s is Q(s,a)=r_M(s,a)+γV(s_μ). When r_M is constant over the actions mapping to s', the induced Q is constant on that class and the MaxEnt-optimal policy is uniform within it. The target π can be recovered only if π happens to be uniform on every successor class, or if r_M happens to reproduce it, but r_M is fixed by the original MDP. Concretely, for M with S={s,s_T}, A={a1,a2}, P(s_T|s,a_i)=1, and r≡0, every extension under Definition 5.1 gives Q(s,a1)=Q(s,a2), so the MaxEnt policy is (1/2,1/2); no choice of |A_1^μ|, |A_2^μ|, or r(s_T^μ) can achieve π=(0.9,0.1). This falsifies Theorem 5.5 and the worst-policy corollary (Corollary 5.7).","section":"Section 5, Definition 5.1; Appendix B, Lemma B.1"},{"comment":"The composition step V(s_μ)=Q(s,a)/P(s'|s,a) is not a valid definition. For a fixed successor class, Q(s,a) must be constant over a for the construction to be realizable, while the quotient depends on a through P(s'|s,a); if two actions in the class have different transition probabilities, the same state s_μ receives conflicting target values. The expression is also undefined for transitions with probability zero, and the proof does not state how such actions are handled. The forward-compatibility lemma cannot absorb this inconsistency, since the target value imposed on s_μ must be single-valued.","section":"Appendix B, Theorem B.3"}],"minor_comments":[{"comment":"The proof refers to 'Definition 4' for the soft value, but no Definition 4 appears in the main text; the definition appears only as Eq. (4) inside the lemma's proof.","section":"Appendix B, Lemma B.1"},{"comment":"The set of extensions is written E_s(M), whereas Definition 5.1 writes E(M,s); the notation should be unified.","section":"Section 5, Notation 5.2"},{"comment":"The theorem statement says 'Let M be an MDP', but Definition 5.1 assumes a continuous action space; the theorem should state the same assumption, or the definition should be extended to discrete actions.","section":"Theorem 5.5"},{"comment":"The sentence 'the simple change of SAC-MaxEnt is not intended as a new efficient algorithm' appears to refer to SAC-AdaEnt and should be corrected.","section":"Section 6.3"}],"recommendation":"reject","confidential_remarks":"The stress-test concern lands directly: the proof of Theorem 5.5 is not merely incomplete; the stated theorem is false under Definition 5.1. A revision that changes Definition 5.1, for example by creating one new state per action rather than per successor, could potentially restore a version of the result, but that is a substantial reconstruction. The current manuscript's central claim is not salvageable by a local proof patch."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the central theorem does not hold as stated. The proof of Lemma B.1 constructs Q(s,a)=α log π(a|s)+v_s for an arbitrary target policy, but Definition 5.1 forces all actions that lead to the same successor to share the same continuation value through a single new state s_μ. So the constructed Q is not realizable. The concrete two-action example in the stress test is correct: with both actions going to the same terminal state and zero rewards, Q(s,a1)=Q(s,a2), so the MaxEnt policy is uniform and no choice of the extension parameters can make it (0.9,0.1). That kills the 'arbitrary policy' claim and with it Corollary 5.7. The line in Theorem B.3 writing V(s_μ)=Q(s,a)/P(s'|s,a) is also invalid when multiple actions share a successor, compounding the problem.\n\nWhat is worth keeping: the toy example in Section 4 is analytically correct and shows the qualitative effect clearly. The empirical Q-landscape comparisons in the control environments are suggestive, and the idea that entropy can mislead at critical states is a reasonable one, consistent with Ahmed et al. 2019 and other prior work. The paper is honest about the trade-off and the adaptive entropy experiment is a nice sanity check, though it is not presented as a new algorithm.\n\nThe soft spots are serious. The main proof is not a minor gap; it fails on a simple deterministic MDP. The paper also does not release code or data, which matters because the experimental claims rely on SAC and PPO implementations. The related work section is adequate, and the paper does not hide its debts. The self-citation issue is minor.\n\nOverall, this is a paper with an interesting observation and a flawed formalization. The qualitative message survives, but the theoretical contribution as written does not. If the authors can fix the realizability condition—perhaps by allowing the extension to add one state per (successor, action) pair or by explicitly allowing reward shaping at s—the theorem might be recoverable for a narrower claim. As is, a serious referee would need to send it back for major revision.\n\nShould we take it to reading group? Maybe, as a case study in how a plausible construction can hide a realizability assumption. But I would not cite it in its current form.","headline":"The main theorem overreaches: the constructed Q-function in Lemma B.1 is not realizable under Definition 5.1, so the arbitrary-policy claim fails; the toy example and experiments are still worth a look.","tokens_in":22872,"tokens_out":3765,"would_cite":false,"duration_ms":34295,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's entropy bifurcation extension can make the maximum-entropy-optimal policy at a chosen state match any arbitrary action distribution — even the worst — while leaving all other states' optimal policies untouched.","keywords":["maximum entropy reinforcement learning","Soft Actor-Critic","policy optimization","entropy bifurcation extension","robustness-optimality tradeoff","continuous control","soft Q-value landscape","entropy trap"],"falsifier":"Take a three-state MDP in which two distinct actions at the targeted state $s$ both transition deterministically to the same successor $s'$, and prescribe a target policy $\\pi(\\cdot|s)$ that assigns these two actions unequal probabilities. In the extension both actions pass through the same inserted state $s_\\mu$, so they are forced to share one soft value; substituting the required equations $Q(s,a) = \\alpha \\log \\pi(a|s) + v_s$ for both actions decides whether the arbitrary-policy clause of Theorem 5.5 holds for such MDPs, because unequal target probabilities make the two equations inconsistent.","tokens_in":21638,"feed_emoji":"🤖","tokens_out":18756,"duration_ms":173732,"temperature":0.7,"pith_summary":"The paper argues that the entropy bonus in maximum-entropy reinforcement learning is not merely an exploration aid that costs a little optimality: at convergence it can flip which policy is optimal, and the flip can be made arbitrarily large. Its main theorem constructs, for any Markov decision process and any chosen state, an entropy bifurcation extension — new branching states with tunable rewards — such that the MaxEnt-optimal policy at that state becomes any prescribed action distribution, even the worst one, while the ordinary optimal policy on every other state is unchanged. The same soft-value mechanism is then used to explain why SAC reliably fails on realistic control tasks such as quadrotor trajectory tracking, high-speed vehicle path following, and quadruped locomotion where PPO succeeds, and why SAC excels on exploration-friendly benchmarks. Understanding the mechanism matters because it separates the genuine exploration benefits of entropy from a convergence-level bias that reward design alone cannot remove.","feed_headline":"One trick makes MaxEnt RL treat any policy as optimal","feed_subtitle":"Entropy-augmented methods like SAC can converge to confidently wrong policies when precise low-entropy action is needed.","key_machinery":"The carrying object is the Entropy Bifurcation Extension (Definition 5.1): for each successor state $s'$ of the targeted state $s$, insert a new state $s_\\mu = \\mu(s')$ whose action space splits into a return interval $A_1^\\mu$ that leads back to $s'$ and a dead-end interval $A_2^\\mu$ that leads to a new terminal state with a tunable reward. Two lemmas do the work. Backward Compatibility (Lemma 5.3) shows that any target distribution $\\pi(\\cdot|s)$ can be matched by setting $Q(s,a) = \\alpha \\log \\pi(a|s) + v_s$, with the constant $v_s$ chosen so the desired soft value is realized and $D_{\\mathrm{KL}}(\\pi \\| \\pi^*_Q) = 0$. Forward Compatibility (Lemma 5.4) shows that the branch widths $|A_1^\\mu|$, $|A_2^\\mu|$ and the terminal reward $r(s_T^\\mu)$ provide enough freedom to realize any target soft value $V(s_\\mu)$ without changing the original successor's value $V(s')$. Together the lemmas let the extension reshape the soft-Q landscape at $s$ — and hence the Boltzmann policy that probability-matches it — independently of all other states.","core_discovery":"The paper's central claim is that in the maximum-entropy RL framework the soft Bellman backup makes the optimal policy itself a function of the soft Q-value landscape: the MaxEnt-optimal policy at a state is the Boltzmann distribution over $Q(s,a)$, so whoever can shape that landscape controls the policy. The paper proves this control can be exercised essentially arbitrarily. Given any MDP, any state $s$, and any desired distribution $\\pi(\\cdot|s)$ over actions, there is an entropy bifurcation extension of the MDP in which $\\pi(\\cdot|s)$ is the MaxEnt-optimal policy at $s$, the soft value $V(s)$ is unchanged, and no other state's optimal policy is altered (Theorem 5.5). Because the construction can be applied at every state independently, the MaxEnt-optimal policy of the extended MDP can realize the worst policy's value while the ground-truth optimal policy retains the best value (Corollary 5.7). The paper emphasizes that this is an effect at convergence of the exact MaxEnt objective, not an artifact of function approximation, exploration noise, or sample efficiency, and it argues with soft-versus-plain Q-value comparisons on learned control tasks that the same bifurcation structure arises naturally when critical states demand precise low-entropy actions and deviations lead to irrecoverable failure.","pith_inferences":["Following the paper's closing suggestion about human-feedback scenarios, the extension reads as a recipe for a stealthy policy attack: an adversary who can add states or shape rewards — for example through a learned reward model — can steer the MaxEnt policy to any target distribution while leaving the return-optimal policy untouched.","A natural repair for the shared-successor gap is to first split shared successors at the targeted state (one copy per action) before applying the extension; if the arbitrary-distribution theorem then holds verbatim, the shared-backup constraint is confirmed as the sole obstruction, and a refined theorem should be tested in that setting.","The soft-versus-plain Q discrepancy that SAC-AdaEnt exploits is measurable online, so the same idea could serve as a monitoring signal in deployed control systems: when the two landscapes diverge sharply at a state, the agent is at risk of an entropy trap and should fall back to plain-value updates or raise the cost of exploration there."],"forward_implications":["At convergence the MaxEnt-optimal policy can be arbitrarily worse than the standard optimal policy: applying the extension at every state makes the MaxEnt optimum realize the worst policy's value $J^-$ while the ground-truth optimum keeps $J^+$ (Corollary 5.7).","The misleading effect is intrinsic to the soft Bellman objective, so it survives perfect function approximation and infinite samples; it is a property of the objective, not of training dynamics.","Control tasks whose critical states require narrow, low-entropy actions (quadrotors, high-speed vehicles, quadrupeds) will systematically favor high-entropy 'mediocre' branches that drift into irrecoverable failure, which the paper argues explains the SAC failures where PPO succeeds.","The same mechanism explains MaxEnt successes: on exploration-friendly tasks such as Hopper, Acrobot, and Obstacle2D, entropy-smoothed Q-landscapes push the policy through risky but promising moves, whereas plain advantage landscapes trap PPO in zero-level sets.","Switching the policy update from soft Q to plain Q where the two landscapes diverge (SAC-AdaEnt) recovers performance in the failing environments, confirming the divergence as the cause; the authors present this as a diagnostic rather than a scalable algorithm."],"supporting_citations":[{"why":"Introduces the maximum-entropy objective and probability-matching optimality notion that the paper shows can be manipulated.","marker":"Ziebart et al., 2008"},{"why":"Defines Soft Actor-Critic; its soft Bellman backup and Boltzmann policy update are the concrete MaxEnt machinery the analysis targets.","marker":"Haarnoja et al., 2018a"},{"why":"Prior study of entropy's impact on policy optimization that the paper extends from landscape smoothing to convergence-level misdirection.","marker":"Ahmed et al., 2019"},{"why":"The robustness rationale for MaxEnt RL that the paper qualifies by exhibiting a regime where entropy misleads rather than protects.","marker":"Eysenbach & Levine, 2021"},{"why":"PPO, the non-MaxEnt baseline that succeeds on the control environments where the paper shows SAC is misled.","marker":"Schulman et al., 2017"},{"why":"Supplies the dynamic bicycle vehicle model used in the Vehicle environment where the misleading soft-Q landscape is demonstrated.","marker":"Kong et al., 2015"},{"why":"Supplies the quadrotor modeling used in the Quadrotor environment where SAC fails and PPO succeeds.","marker":"Rubí et al., 2020"}],"fun_headline_variants":["MaxEnt RL optimal policy is forgeable via Q-value shaping","Entropy maximization can be tricked into favoring any action","Soft Bellman backup makes any policy look MaxEnt-optimal","MaxEnt RL misleads when tasks demand low-entropy precision","Any policy can be optimal under MaxEnt RL by design"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The construction assumes that every action at the targeted state can be routed through its own new branching state with an independently tunable reward; when several actions share the same successor state, they share one backup value, so a prescribed policy that treats those actions differently may not be realizable.","fun_headline_variants_meta":{"raw":{"variants":["MaxEnt RL optimal policy is forgeable via Q-value shaping","Entropy maximization can be tricked into favoring any action","Soft Bellman backup makes any policy look MaxEnt-optimal","MaxEnt RL misleads when tasks demand low-entropy precision","Any policy can be optimal under MaxEnt RL by design"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000399,"raw_usage":{"total_tokens":2082,"prompt_tokens":935,"completion_tokens":1147,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1061}},"tokens_in":551,"tokens_out":1147,"duration_ms":12565,"temperature":1.0,"reasoning_tokens":1061,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:14:18.680158+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a three-state MDP in which two distinct actions at the targeted state $s$ both transition deterministically to the same successor $s'$, and prescribe a target policy $\\pi(\\cdot|s)$ that assigns these two actions unequal probabilities. In the extension both actions pass through the same inserted state $s_\\mu$, so they are forced to share one soft value; substituting the required equations $Q(s,a) = \\alpha \\log \\pi(a|s) + v_s$ for both actions decides whether the arbitrary-policy clause of Theorem 5.5 holds for such MDPs, because unequal target probabilities make the two equations inconsistent.","supporting_citations":[{"cited_title":"D., Maas, A","cited_arxiv_id":null,"evidence_quote":"Introduces the maximum-entropy objective and probability-matching optimality notion that the paper shows can be manipulated."},{"cited_title":"Understanding the impact of entropy on policy optimization","cited_arxiv_id":null,"evidence_quote":"Prior study of entropy's impact on policy optimization that the paper extends from landscape smoothing to convergence-level misdirection."},{"cited_title":"Kinematic and dynamic vehicle models for autonomous driving control design","cited_arxiv_id":null,"evidence_quote":"Supplies the dynamic bicycle vehicle model used in the Vehicle environment where the misleading soft-Q landscape is demonstrated."}],"review_version":1}