{"id":"e7074655-f290-4c33-acdb-0b3b296555eb","arxiv_id":"2607.17316","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"With a chance/choice distinction, independence of irrelevant alternatives and monotonicity force any choice-rule/value pair to be Boltzmann softmax with a Shannon-entropy bonus and soft Bellman equation.","lead":"Softmax choice—pick actions with probability proportional to the exponential of their estimated value—is shown to follow logically from two principles: separating environmental randomness from an agent's own choice freedom, and requiring independence of irrelevant alternatives plus monotonicity. This gives a normative foundation for the entropy bonus in modern RL and shows when it should not be used, such as with redundant similar options.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bellman-extension assumption is load-bearing: conclusion (iii) does not follow from Axioms 1–4 alone, so the 'unique characterization' is conditional on an unproven structural assumption.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: the Bellman extension is not derived and is required for conclusion (iii). I agree this is the most critical point. The paper is honest about the assumption, explicitly flagging it in Section 3.1, and the main derivation of the entropy bonus (Step 2) is sound and independent of the extension. However, the central claim of a 'complete axiom-to-algorithm path' is somewhat overstated because the soft Bellman equation is not a consequence of the stated Axioms 1–4 on their own; it also requires the unproven extension. This does not invalidate the paper's contribution—the theorem is true conditional on the extension, and the paper openly acknowledges the limitation—but it explains the moderate confidence. The proposed concrete test would settle whether the extension is genuinely necessary or whether it can be derived from simpler consistency requirements; until then, the reader's ACCEPT verdict remains appropriate, with the caveat that the characterization is conditional. No ad hominem, no theatrical language: the concern is about the logical structure of the proof, specifically the transition from Step 2 to Step 3.","tokens_in":15301,"tokens_out":10157,"duration_ms":106845,"concrete_test":"Construct a minimal finite MDP (e.g., two states, three actions per state) in which the successor choice-node value is defined by the hard-max operator, V'(s′) = max_a Q(s′,a), while the current-state choice rule is still Boltzmann in the resulting Q-values and the menu value is V(s) = (1/β)LSE(βQ(s,·)). Check whether (a) Axioms 1–2 hold for all deterministic continuation policies under this alternative recursion, and (b) Axioms 3–4 still hold at every choice node. If such a model is consistent, then Axioms 1–4 alone do not force the soft Bellman equation; the extension is doing essential work. If it is inconsistent, identify which axiom is violated and show that the extension is actually forced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that IIA and monotonicity uniquely determine the soft Bellman equation—depends on the Bellman-extension assumption stated in Section 3.1: 'Here we assume that the Bellman relation extends from deterministic continuation policies to successor choice nodes.' This extension is not derived from Axioms 1–4. It is used in Step 3 of Theorem 3.1 to substitute V = (1/β)LSE(βQ) into Q(s,a) = R(s,a) + γ(s,a)E[V(s′)], thereby yielding the soft Bellman equation. Without the extension, Axioms 1–2 only determine Q for deterministic continuation policies; the value of a successor choice node (where the agent will later randomize) is not defined by the VNM/DC axioms. Step 2 does derive V = LSE(βQ) as the value of a menu given Q-values, but it does not establish that the Q-values in different states are coupled by R and γ through this V. That coupling is exactly what the extension supplies. Since the extension is stated in prose rather than as a numbered axiom, the theorem's 'if and only if' is not a characterization purely in terms of the qualitative axioms. The entropy-bonus derivation (Step 2) is independent of this concern, but the soft Bellman equation—a headline contribution—is conditional on an extra structural assumption that is neither derived nor given a normative justification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an axiomatic derivation of entropy-regularized (Boltzmann) policies and the soft Bellman equation in finite MDPs. The key modeling move is to restrict VNM Independence to environmental lotteries over deterministic base prospects, reserving IIA and monotonicity (in both Policy and Value versions) for the agent's choice rule at decision nodes. Theorem 3.1 claims these axioms uniquely imply the softmax policy, the entropy-regularized policy objective, and the soft Bellman equation; the proof proceeds in three steps (Cauchy functional equation, menu-value functional equations, then Bellman recursion). The paper then studies consequences: divergence of the soft Bellman equation under γ>1 and its resolution by horizon continuity, monotonicity of actual return in β, and a numerical demonstration that Shannon entropy is the only IIA-consistent member of the Tsallis family. It closes with a cross-disciplinary synthesis and a normative discussion of when IIA is appropriate.","tokens_in":15696,"tokens_out":10230,"duration_ms":101097,"significance":"If the characterization were fully rigorous, it would be a valuable contribution: it would supply a missing first-principles justification for softmax in RL, clarify the normative content of the entropy bonus as an 'option premium,' and connect RL to a large body of choice-theoretic and econometric results. The paper is unusually transparent about the strength of its axioms, especially Value IIA, and the Step 2 derivation of the Shannon entropy bonus from menu-level functional equations is a genuine contribution. The discussion of generalized discounting and the β0 threshold is also suggestive. However, the central theorem depends on an extra structural assumption about the Bellman relation that is not derived from the stated axioms, which currently limits the force of the uniqueness claim.","major_comments":[{"comment":"The assumption that 'the Bellman relation extends from deterministic continuation policies to successor choice nodes' is stated in prose and is not derived from Axioms 1–4. Step 3 uses it to write Q(s,a)=R(s,a)+γ(s,a)E[V(s′)] with V(s′) the choice-node value, and then to substitute V=(1/β)LSE(βQ). Without this extension, Axioms 1–2 only determine Q for deterministic continuations, and Axioms 3–4 only determine V as a function of an arbitrary Q-vector at a single menu. The theorem's 'if and only if' is therefore conditional on an unproved structural assumption, and the abstract's claim that IIA and monotonicity uniquely determine the soft Bellman equation is not supported. Please either prove the extension from Axioms 1–2 or state it as an explicit numbered axiom with its own normative justification.","section":"Section 3.1 / Step 3 of Theorem 3.1"},{"comment":"The proof of Step 2 assumes that submenu limits exist and are finite, and specifically that f(p)→0 as p→0, in order to conclude h(p)→0 at the boundary and hence h≡0. This assumption is not a consequence of Axioms 3 and 4 as stated (monotonicity alone does not give continuity at the boundary of the probability simplex). It is load-bearing for the uniqueness of the Shannon entropy among solutions of the functional equations (1)–(3). If this regularity is needed, it should be elevated to an explicit axiom or proved from finiteness of the action set plus the other axioms; otherwise non-Shannon solutions may survive.","section":"Footnote 1 / Step 2 of Theorem 3.1"}],"minor_comments":[{"comment":"Typo: 'V on Neumann-Morgenstern' should read 'von Neumann–Morgenstern.'","section":"Section 1"},{"comment":"The footnote markers (∗, †) are not fully defined in the caption; in particular, the Thompson-sampling row uses '✓∗' in two columns but the footnote only explains the conditions partially. Please clarify.","section":"Table 1"},{"comment":"The y-axis label 'std of log (a)/(b)' should be rendered as 'std of log(π(a)/π(b))' and the simulation details (number of draws, treatment of zero-probability draws) should be stated in the figure caption.","section":"Figure 4.3"},{"comment":"The proof suppresses the start state and asserts that πα is optimal from every start state at once. This is true for the regularized MDP but should be stated explicitly, since the reader might otherwise wonder whether the envelope argument requires a single start state.","section":"Proof of Proposition 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a reinforcement learning journal. The main gap is the Bellman-extension assumption in Section 3.1: it is load-bearing for the soft Bellman equation but is not derived from Axioms 1–4. If the authors can either prove it or make it a transparent axiom, I would support acceptance; the reader's accept verdict is understandable, but the current 'if and only if' claim is stronger than what the axioms alone support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper delivers a real, new axiomatic path from IIA and monotonicity to the Boltzmann policy and the entropy bonus, and the proof is mostly sound. The one thing to keep in mind: the soft Bellman equation itself rests on an explicit extension assumption — the Bellman relation is assumed to carry over from deterministic continuation policies to successor choice nodes — that is not derived from the axioms and is doing real work. The paper is honest about this, but it means the headline \"uniquely determines the soft Bellman equation\" is conditional.\n\nWhat's genuinely new: Step 2, where Value IIA plus monotonicity forces the entropy bonus to be exactly the Shannon entropy, without assuming an additive perturbed-utility form. That's a real improvement over Fudenberg, Iijima, and Strzalecki, who need the additive structure. Step 1 is classical Luce, but the way the paper assembles the pieces into a single axiom-to-algorithm path is new. The RL consequences — the β0 threshold under amplifying discounting and its resolution via horizon continuity — are a nice bonus, not the main event. The discussion of IIA failure modes (red bus/blue bus, RLHF perverse incentives) is genuinely useful and well-cited.\n\nSoft spots: the Bellman-extension assumption is the big one. The theorem's conclusion (iii) only follows because Step 3 substitutes V = (1/β)LSE(βQ) into an already-extended Q(s,a) = R(s,a) + γ(s,a) E[V(s')]. Without that extension, Axioms 1 and 2 only define Q for deterministic continuations. The paper states the extension in prose rather than as a numbered axiom; I'd want the referee to push on whether it can be justified from more basic principles, or at least be made an explicit axiom with its normative content visible. The abstract oversells slightly by not mentioning this condition.\n\nTwo minor things: the continuity argument in Step 2 is compressed — it's correct, but a referee should check that the jump argument is fully spelled out. Proposition 4.3's envelope proof is also terse, though standard. Value IIA is strong, but the paper acknowledges this and provides equivalent formulations (marginal consistency, additive form), which helps.\n\nWho should read this: RL theorists and anyone who uses entropy regularization and wants to understand what softmax is actually assuming. Also people working on RLHF, since the IIA failure-mode discussion is directly relevant.\n\nMy recommendation: send it to peer review. It deserves a serious referee. The central argument holds up; the Bellman-extension assumption is the thing to scrutinize, not a reason to desk reject.","headline":"Novel axiom-to-softmax path with a solid core, but the soft Bellman equation rides on an explicit, unproven extension assumption that the abstract glosses over.","tokens_in":16130,"tokens_out":6400,"would_cite":true,"duration_ms":60125,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["91B16","90C40","91A10"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that, once von Neumann–Morgenstern independence is restricted to environmental randomness, the combination of independence of irrelevant alternatives and monotonicity uniquely forces the softmax (Boltzmann) policy, the ent","keywords":["Axiomatic RL","Decision Theory","Stochastic Policies","Independence of Irrelevant Alternatives","Entropy Regularization","Boltzmann Rationality","Soft Bellman Equation"],"falsifier":"Numerically solve the two functional equations from Step 2 of the proof—f(p)-f(1-p)=(1/β)log(p/(1-p)) and f(p/(1-t))+f(t)=f(t/(1-p))+f(p) for p+t<1—for continuous, monotone f with f(0)=0. If a solution other than f(p)=-(1/β)log(1-p) exists, the axiomatic system admits non-Boltzmann policies and Theorem 3.1 is false; a reader can test this directly with a simple solver.","tokens_in":15213,"feed_emoji":"🎲","tokens_out":8947,"duration_ms":81390,"temperature":0.7,"pith_summary":"This paper tries to prove that the softmax (Boltzmann) policy, the backbone of modern reinforcement learning, is not just a convenient or robust heuristic but the unique rational response to a precise set of axioms. The move is to split randomness into two kinds: environmental chance, which is evaluated by von Neumann–Morgenstern expected utility from deterministic plans ('base prospects'), and the agent's own choice, which is governed by independence of irrelevant alternatives and monotonicity. If those axioms are accepted, the paper's main theorem derives the softmax policy, the entropy-regularized objective, and the soft Bellman equation from scratch, giving a first-principles justification for why the entropy bonus appears. The paper also shows what changes if the axioms are rejected—correlated action sets can break IIA, producing the red-bus/blue-bus failure mode—and draws concrete reinforcement-learning consequences like a minimum rationality temperature under amplifying discounting. A sympathetic reader should care because the paper converts a default engineering choice into a testable normative commitment, and makes explicit which axis of the problem (chance vs. choice) is doing the moral work.","feed_headline":"IIA axioms alone force Boltzmann policies and soft Bellman equation","feed_subtitle":"Once VNM independence is limited to environmental luck, menu independence and monotonicity pin down the entropy bonus.","key_machinery":"The load-bearing object is the chance/choice separation. A base prospect is a deterministic plan with only environmental randomness; VNM Independence applies there. At choice nodes, two axioms are imposed: Policy IIA requires the ratio π(a|q)/π(b|q) to depend only on q_a−q_b, and Value IIA requires each action's selection probability to be a function of the loss in menu value when that action is removed. Monotonicity makes those functions invertible. The proof's engine is the functional equation that emerges from evaluating a three-action menu by adding actions in different orders; its unique solution is f(p)=-(1/β)log(1-p), which converts the value of any menu into (1/β)LSE(βQ). Marginal co","core_discovery":"The paper's central claim is Theorem 3.1: in a finite MDP with at least three actions and contraction discounting, an agent whose preferences over environmental lotteries satisfy VNM independence, whose Bellman relation is extended to successor choice nodes, and whose choice rule and value function satisfy IIA (both policy-level and value-level) and monotonicity must have π(a|s) ∝ exp(βQ(s,a)), must choose π to maximize Σ_a π_a Q(s,a) + (1/β)H(π), and must have V(s) = (1/β) LSE(βQ(s,·)) with Q(s,a) = R(s,a) + γ(s,a) E[V(s′)]. In other words, the entropy bonus is not an ad hoc regularizer but the exact 'option premium' an agent pays for being able to choose among alternatives rather than faci","pith_inferences":["The axiomatization yields a practical audit: check whether log(π(a)/π(b)) changes when other actions are added or removed; violations signal whether IIA is appropriate for the domain, which is a testable property of deployed policies.","The same chance/choice split could justify hierarchical choice models such as nested logit when actions are correlated; the paper points to this as open, and it would be the natural fix for the near-duplicate problem in language-model decoding.","If the Gumbel-noise story and the information-cost story are equivalent to this axiomatic system under the stated conditions, then those practical justifications are not independent heuristics but consequences of the same axioms—suggesting the Boltzmann form is overdetermined.","Because only the product β·reward-scale is identified, any claim about 'the right temperature' for stochastic policies is empty without fixing the reward scale; hyperparameter searches can be reduced to one dimension."],"forward_implications":["If the axioms are accepted, any agent whose action preferences are menu-independent must be a Boltzmann agent; the only free parameter is the inverse temperature β.","Shannon entropy is the unique entropy that satisfies Value IIA; Tsallis, sparsemax, and other regularizers are excluded by the axioms, not just by convenience.","The choice between the soft and the hard Bellman equation is a normative choice: whether the agent values its own ability to choose.","With state-action dependent discount factors larger than one, the soft Bellman equation can fail to have a finite fixed point below a threshold β0; horizon continuity, which keeps all discounted transition matrices contractive, removes the threshold and guarantees convergence for all β.","Actual unregularized return is non-decreasing in β, so increasing deliberateness (higher β) never hurts true performance in finite discounted MDPs."],"fun_headline_variants":["Entropy bonus is not ad hoc: IIA forces it","Softmax is inevitable: axioms pin it down","Entropy premium: the cost of being able to choose","Rationality with choice: IIA yields softmax policies","Why softmax? Because IIA demands it"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper explicitly assumes, rather than derives, that the Bellman relation extends from deterministic continuation policies to successor choice nodes (Section 3.1), so that Q(s,a)=R(s,a)+γ(s,a)E[V(s′)] with V evaluated at the choice-node value; without this extension, the derivation of the soft Bellman equation does not follow from Axioms 1 and 2 alone, and the theorem's conclusion (iii) lacks support from the stated axioms.","fun_headline_variants_meta":{"raw":{"variants":["Entropy bonus is not ad hoc: IIA forces it","Softmax is inevitable: axioms pin it down","Entropy premium: the cost of being able to choose","Rationality with choice: IIA yields softmax policies","Why softmax? Because IIA demands it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1630,"prompt_tokens":810,"completion_tokens":820,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":742}},"tokens_in":554,"tokens_out":820,"duration_ms":7469,"temperature":1.0,"reasoning_tokens":742,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T18:21:37.885713+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Numerically solve the two functional equations from Step 2 of the proof—f(p)-f(1-p)=(1/β)log(p/(1-p)) and f(p/(1-t))+f(t)=f(t/(1-p))+f(p) for p+t<1—for continuous, monotone f with f(0)=0. If a solution other than f(p)=-(1/β)log(1-p) exists, the axiomatic system admits non-Boltzmann policies and Theorem 3.1 is false; a reader can test this directly with a simple solver.","supporting_citations":[],"review_version":1}