{"id":"6094dd03-e14d-43d2-8085-f39bf454e7b7","arxiv_id":"2501.09770","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"EVAL learns the optimal policy for entropy-regularized average-reward MDPs by training neural networks to approximate the dominant eigenvector of a tilted transition matrix, with a variant that recovers the unregularized solution.","lead":"This paper introduces a deep reinforcement learning algorithm that solves a version of the average-reward problem with entropy regularization by learning the main eigenvector of a tilted transition matrix. It also shows how to remove the regularization to recover the standard average-reward objective, and reports better stability and speed than DQN and soft Q-learning on small control tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main experiments fix θ=0, so the learned u is not the Perron eigenvector of Eq. (13); the reported results do not validate EVAL as described, and the optimality of the θ=0 variant is unjustified.","rationale":"The reader's weakest_assumption already identified the θ=0 contradiction as part of the core weakness. I agree that this is the most load-bearing concern because it directly undermines the empirical validation of the central claim. The paper's derivation of the update rules is explicitly tied to the eigenvalue equation (13), which only holds at the true Perron exponent θ. The Appendix reveals that all main experiments freeze θ at zero, so the trained objective is not the one analyzed. While the direction of the Perron eigenvector is invariant under scalar scaling of the tilted matrix, the θ=0 variant is not guaranteed to recover that direction: the fixed-point equation u = e^{βr}E[u] generally has no positive solution, and the bootstrapped target can cause scale divergence. The paper provides no analysis of this variant, so the empirical success, if real, is unexplained and may rely on undocumented properties of the environments or network architecture. The lack of a convergence proof for the original θ-updated algorithm is also significant, but it is explicitly deferred in the Limitations section and therefore is a transparent limitation rather than a hidden contradiction. The θ=0 issue is more damaging because it means the experiments do not test the algorithm as written. Thus the conditional verdict is appropriate: the theoretical idea is coherent in the tabular case, but the authors must reconcile the experiments with the proposed update rules, either by running with the θ update or by providing a rigorous justification for the θ=0 simplification. No verdict change is needed beyond the reader's conditional acceptance.","tokens_in":17841,"tokens_out":13323,"duration_ms":139243,"concrete_test":"Re-run all three classic-control benchmarks with the θ update of Eq. (16) enabled, holding all other hyperparameters fixed; if the θ-updated algorithm diverges, performs substantially worse, or produces unstable u norms, the central claim lacks empirical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the TD update (Eqs. 14–16) converges to the Perron eigenvector u of the tilted matrix (Eq. 8), and that the policy (Eq. 10) built from u solves the ERAR MDP. This hinges on θ in Eq. (15) being the true entropy-regularized reward rate, because Eq. (13) is an eigenvector equation only at that θ. The Appendix ('Experimental Details') states: 'The ERAR rate, θ is kept frozen at zero (except for irreducible tabular dynamics).' With θ=0, the target becomes e^{βr}E[u], and the fixed-point equation u = e^{βr}E[u] is not Eq. (13) unless the Perron root e^{βθ} happens to equal 1. For a positive tilted matrix, the Perron-Frobenius eigenvector is unique; if e^{βθ}≠1, the equation u = e^{βr}E[u] has no positive fixed point (the only finite fixed point is u=0). Thus the algorithm actually trained in the experiments is not the algorithm derived from the eigenvalue problem, and the reported policy performance cannot be attributed to the theory in Lemma 1. The paper provides no convergence analysis for the θ=0 variant, nor any argument that its policy still matches the ERAR optimal policy. This is not a mere implementation detail: it changes the fixed point of the learning dynamics and invalidates the theoretical grounding of the experimental results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EVAL, an off-policy algorithm for entropy-regularized average-reward reinforcement learning (ERAR MDPs) based on stochastic approximation of the dominant eigenvector u of a tilted transition matrix. The optimal policy and the entropy-regularized reward rate are recovered from u via Eq. (10) and the Perron root. The paper also introduces a posterior policy iteration (PPI) scheme to recover the unregularized average-reward policy, and reports experiments on classic control environments comparing EVAL and EVAL+PPI against DQN and Soft Q-Learning. The theoretical derivation follows Arriojas et al. (2023b); the algorithmic contribution is a deep-RL implementation with a temporal-difference update for u (Eqs. 14--15) and an online estimate of the reward rate θ (Eq. 16).","tokens_in":18121,"tokens_out":8095,"duration_ms":77332,"significance":"If the proposed stochastic approximation converges, the tilted-eigenvector formulation is an elegant and principled way to solve ERAR MDPs without a discount factor, and the PPI trick provides a potentially useful route to unregularized average-reward control. The paper is clearly written, and the authors state that the code is publicly available. However, the experimental evaluation as reported does not test the algorithm as described: the appendix states that θ is frozen at zero in all non-tabular benchmarks, which changes the fixed point of the learning update and invalidates the theoretical grounding of the experiments. This is a serious gap that must be addressed before the empirical claims can be accepted.","major_comments":[{"comment":"The appendix states 'The ERAR rate, θ is kept frozen at zero (except for irreducible tabular dynamics).' This directly contradicts Algorithm 1 (lines 17 and 21) and Eq. (16), which update θ online. With θ fixed at zero, the TD target in Eq. (15) becomes e^{βr} E[u], and the corresponding fixed-point equation is u = e^{βr} E[u]. This is not the eigenvector equation (13) unless e^{βθ} = 1; for a positive tilted matrix with Perron root e^{βθ} ≠ 1, the only nonnegative fixed point is u = 0. Therefore the reported benchmark results do not validate EVAL as presented. The authors should either rerun all experiments with the θ update active, or provide a separate theoretical justification for the θ = 0 variant and show that its policy still solves the ERAR MDP.","section":"Appendix, Experimental Details; Eqs. (15)--(16); Algorithm 1"},{"comment":"No convergence analysis is provided for the coupled stochastic approximation of u and θ, and the paper explicitly defers this to future work. Since the central claim is a 'solution' to deterministic ERAR MDPs with general function approximators, the paper should at least state the conditions under which the updates converge (for example, via two-timescale stochastic approximation with appropriate step-size schedules) or temper the claim to a heuristic algorithm with empirical support. Without this, the theoretical grounding of the function-approximation setting remains incomplete.","section":"Limitations and Future Work; Proposed Algorithms, Eqs. (14)--(16)"},{"comment":"The paper claims to provide 'the first solution to deterministic ERAR MDPs in continuous state spaces by use of function approximation.' Given that the core Lemma 1 is imported from an overlapping-author publication (Arriojas et al. 2023b) and no new convergence result is proven here, the novelty is the algorithmic adaptation plus the empirical study. This is valuable, but the wording overstates the theoretical contribution; please scope the claim accordingly, especially given the issue in the first major comment.","section":"Introduction, Main Contributions; Conclusion"}],"minor_comments":[{"comment":"Line 9 of Algorithm 1 samples actions from π0, but the text in 'Solution to ERAR-MDP' says the replay buffer is collected using the learnt policy (Eq. (10)). Please align the pseudocode with the text, as is done in Algorithm 3.","section":"Algorithm 1"},{"comment":"The θ update averages the ratio e^{βr} E_{a'~π0} u(s',a') / u(s,a) over the batch. Although u uses a softplus output so the denominator is positive, it can be very small and cause large variance or overflow; consider clipping the ratio or using a log-space update to improve numerical stability.","section":"Eq. (16)"},{"comment":"The caption states that the final two columns show hyperparameters specific to PPI, but the table appears to use the same column structure as the preceding hyperparameter table; please clarify which columns are PPI-specific.","section":"Experimental Details, Table 2"},{"comment":"The text refers to 'EVAL-PPI' while Figure 2 and Algorithm 3 use 'EVAL+PPI'; please standardize the notation throughout the manuscript.","section":"Experiments"}],"recommendation":"major_revision","confidential_remarks":"The central issue is that the experimental appendix disables the very mechanism (the θ update) that distinguishes the proposed method from a heuristic variant. The authors should be asked either to rerun the benchmarks with θ updated according to Eq. (16) or to provide a formal analysis of the frozen-θ variant. The self-citation of Lemma 1 from Arriojas et al. (2023b) is acceptable as a building block, but the 'first solution' claim should be calibrated once the convergence and experimental issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the paper's real contribution is a DQN-style neural implementation of the eigenvalue characterization from Arriojas et al. 2023b, plus the PPI loop for unregularized average reward. The tabular derivation is clean, and the experimental comparison against DQN and SQL on classic control is reasonably executed with tuned hyperparameters and multiple seeds. The 'spectral gap discount factor' analysis in Figure 1 is a nice insight. If you work on average-reward RL, the idea is worth knowing.\n\nBut there's a load-bearing inconsistency in the experiments. Algorithm 1 and Eq. (16) update θ, but the appendix states θ is frozen at zero (except irreducible tabular dynamics). With θ=0, Eq. (13) becomes u = e^{βr} E[u], which is not the Perron eigenvector equation unless e^{βθ}=1. For the positive tilted matrix, the unique Perron eigenvector satisfies a different equation, and the θ=0 variant has no positive fixed point when e^{βθ}≠1 (u=0 is the only finite one). The stress-test note gets this right. That means the reported benchmark results do not actually validate EVAL as written; they validate a different algorithm whose optimality is not established. This isn't a minor implementation detail; it changes the fixed point of the learning dynamics. The paper needs to either justify the θ=0 simplification (e.g., show that in these environments the optimal θ is near zero, or that the policy is insensitive to θ) or run the full update.\n\nOther soft spots: no convergence analysis for the stochastic approximation — the paper defers it, which is honest but leaves the central claim unproven. The novelty is mostly in the algorithmic combination; Lemma 1 is from overlapping-author prior work, and PPI from Rawlik. That's fine if credited, and it is.\n\nThe experiments show competitive sample efficiency, and the PPI extension to unregularized average reward is clever. But the theta=0 issue means the empirical support is conditional. For a serious referee, this should be major revision with a clear reconciliation of experiments and algorithm. It is not a desk reject — the idea has legs and the tabular foundation is sound.\n\nRecommendation: send to peer review, but insist on addressing the theta=0 gap and adding statistical rigor (more seeds, confidence intervals, and ideally a convergence check).","headline":"A promising neural extension of the ERAR eigenvalue framework, but the experiments freeze θ=0 and therefore don't validate the algorithm as written.","tokens_in":18682,"tokens_out":3036,"would_cite":true,"duration_ms":26573,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EVAL learns the entropy-regularized average-reward policy by stochastic approximation of the dominant eigenvector of a tilted transition matrix, and a posterior policy iteration loop recovers the unregularized average-reward optimum…","keywords":["average-reward reinforcement learning","entropy regularization","tilted transition matrix","Perron eigenvector","posterior policy iteration","function approximation","off-policy learning","large deviation theory"],"falsifier":"Run EVAL on a deterministic continuing MDP whose ERAR solution can be computed exactly (for instance a small discretized chain whose tilted matrix can be diagonalized), train it to convergence, and compare the learned $u(s,a)$, $\\theta$, and greedy policy against the exact Perron eigenvector, eigenvalue, and ERAR-optimal policy; a mismatch in $\\theta$ or a policy that fails the ERAR Bellman equation would refute the claim, and repeating the test with $\\theta$ unfrozen would settle whether the frozen-$\\theta$ experiments changed the fixed point.","tokens_in":17613,"feed_emoji":"🧮","tokens_out":11886,"duration_ms":107254,"temperature":0.7,"pith_summary":"EVAL is an off-policy algorithm that solves the entropy-regularized average-reward (ERAR) reinforcement-learning problem in continuous state spaces using neural-network function approximation, extending a tabular large-deviation framework in which the optimal policy and reward rate are read off from the dominant eigenvector of a tilted transition matrix. The paper claims that the same eigenvector equation can be learned stochastically by a TD-style update with no discount factor, and that iterating the prior policy (Posterior Policy Iteration, PPI) recovers the unregularized average-reward optimum without temperature annealing. If correct, this gives value-based average-reward RL a practical, DQN-compatible implementation and a principled way to choose a discount factor from the spectral gap of the tilted matrix. The paper supports the claim with classic-control experiments showing better reward and lower variance than DQN and soft Q-learning, including a continuing CartPole task in which EVAL+PPI keeps the pole balanced for very long horizons.","feed_headline":"Eigenvector learning solves average-reward RL without discounting","feed_subtitle":"No discount factor, no annealing: EVAL learns the optimal average-reward policy by tracking an eigenvector.","key_machinery":"The load-bearing object is the tilted transition matrix $\\tilde P_{(s',a'),(s,a)}=p(s'|s,a)\\pi_0(a'|s')e^{\\beta r(s,a)}$, whose Perron root $e^{\\beta\\theta}$ and left eigenvector $u(s,a)$ determine the entropy-regularized average reward rate, the differential value function $Q(s,a)=\\beta^{-1}\\log u(s,a)$, and the optimal policy $\\pi(a|s)\\propto \\pi_0(a|s)u(s,a)$. EVAL carries this identity into a neural-network stochastic approximation of the fixed-point equation $u(s,a)=e^{\\beta(r(s,a)-\\theta)}\\mathbb{E}[u(s',a')]$, using two online networks, two lagging target networks, and a batch update for $\\theta$. The PPI loop treats this solver as a subroutine and iteratively replaces the prior policy with the current optimal-policy estimate, which the paper argues converges to the unregularized average-reward solution without temperature annealing.","core_discovery":"The paper's central discovery is that the entropy-regularized average-reward problem is equivalent to a Perron eigenproblem for the tilted transition matrix $\\tilde P_{(s',a'),(s,a)} = p(s'|s,a)\\,\\pi_0(a'|s')\\,e^{\\beta r(s,a)}$, and that this equivalence can be turned into a stochastic-approximation learning rule rather than remaining a tabular identity. The eigenvector equation is written as $u(s,a)=e^{\\beta(r(s,a)-\\theta)}\\,\\mathbb{E}_{s'\\sim p,\\,a'\\sim\\pi_0}[u(s',a')]$, with $u$ parameterized by two deep networks, trained by minimizing the squared TD error in Eq. (14) against a lagging target network, while the reward rate $\\theta$ is updated from the batch average in Eq. (16). The paper then shows that replacing the prior policy $\\pi_0$ with the current estimate of the optimal policy, at a fixed schedule (posterior policy iteration), yields the greedy average-reward policy without an annealing schedule; it reports that EVAL and EVAL+PPI compare favorably with DQN and soft Q-learning on CartPole, Acrobot, and MountainCar. It asserts that this is the first solution to deterministic ERAR MDPs in continuous state spaces via function approximation.","pith_inferences":["Because the experiments freeze the reward-rate estimate at $\\theta=0$, the empirical results are compatible with a fixed point slightly different from the stated ERAR objective; an ablation that unfreezes $\\theta$ and measures the true entropy-regularized rate would clarify what the method actually optimizes.","If the eigenvector view is right, the same tilted matrix could serve as a diagnostic unifying discounted and average-reward objectives, for instance by estimating the spectral gap online to set an effective horizon without tuning $\\gamma$.","PPI's convergence guarantee assumes each ERAR subproblem is solved before the prior is updated, while the experiments update the prior on a fixed schedule; varying the subproblem solve budget would show how much of the success depends on the schedule.","The method is restricted to deterministic transitions as presented; taking up the paper's suggested iterative reward/dynamics biasing to handle stochastic dynamics would be a natural stress test."],"forward_implications":["Average-reward RL becomes a value-based, off-policy problem solvable with the same replay-buffer and target-network machinery as DQN, so long-horizon continuing tasks no longer require picking a discount factor.","EVAL+PPI solves the unregularized average-reward MDP without discounting or an annealing schedule, by iterating the prior policy toward the current optimal-policy estimate.","The spectral gap of the tilted matrix gives a principled discount factor: it marks where discounted soft Q-learning has essentially converged to the average-reward solution, so increasing $\\gamma$ further no longer pays.","Learning the normalized left eigenvector yields a centered differential value function, avoiding the arbitrary additive constant that many average-reward algorithms carry."],"supporting_citations":[{"why":"Supplies the tilted-matrix eigenvector solution to ERAR MDPs and the tabular update equations (Eqs. 26–27) that EVAL turns into a neural-network stochastic approximation.","marker":"(Arriojas et al. 2023b)"},{"why":"Provides Theorem 4, the posterior policy iteration result that EVAL+PPI uses to recover the unregularized greedy average-reward policy.","marker":"(Rawlik, Toussaint, and Vijayakumar 2012)"},{"why":"Gives the PPI convergence argument and the observation that iterating the prior policy reaches the $\\beta\\to\\infty$ solution without annealing.","marker":"(Rawlik 2013)"},{"why":"Establishes the control-as-inference identification of the soft value function with the log-probability of an optimality-conditioned trajectory.","marker":"(Levine 2018)"},{"why":"Motivates the twin/target-network averaging scheme that stabilizes the eigenvector TD updates.","marker":"(Van Hasselt, Guez, and Silver 2016)"},{"why":"Supplies the DQN architecture, replay buffer, and target-network machinery that EVAL's implementation is built on and that serves as a baseline.","marker":"(Mnih et al. 2015)"},{"why":"Defines soft Q-learning, the entropy-regularized discounted baseline EVAL is compared against in the classic-control experiments.","marker":"(Haarnoja et al. 2017a)"},{"why":"Clarifies the centering problem for average-reward value functions, which EVAL's normalized eigenvector solution avoids.","marker":"(Wan, Naik, and Sutton 2021)"},{"why":"Provides the baseline DQN implementation and tuned hyperparameters used for the experimental comparison.","marker":"(Raffin et al. 2021)"}],"fun_headline_variants":["No discount factor: eigenvector method learns average-reward RL","Average-reward RL via eigenvector, no discounting","Eigenvector algorithm finds average-reward policy without discounting","EVAL: eigenvector learning for average-reward RL without discounts","Top eigenvector yields average-reward optimal policy, no discount"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the neural-network TD updates converge to the dominant eigenvector of the tilted matrix even with function approximation, and that freezing the reward-rate estimate at zero in the experiments still leaves the ERAR-optimal fixed point; the paper supplies no convergence proof and defers it to future work.","fun_headline_variants_meta":{"raw":{"variants":["No discount factor: eigenvector method learns average-reward RL","Average-reward RL via eigenvector, no discounting","Eigenvector algorithm finds average-reward policy without discounting","EVAL: eigenvector learning for average-reward RL without discounts","Top eigenvector yields average-reward optimal policy, no discount"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001449,"raw_usage":{"total_tokens":5854,"prompt_tokens":982,"completion_tokens":4872,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":4786}},"tokens_in":598,"tokens_out":4872,"duration_ms":32698,"temperature":1.0,"reasoning_tokens":4786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:10:46.583768+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EVAL on a deterministic continuing MDP whose ERAR solution can be computed exactly (for instance a small discretized chain whose tilted matrix can be diagonalized), train it to convergence, and compare the learned $u(s,a)$, $\\theta$, and greedy policy against the exact Perron eigenvector, eigenvalue, and ERAR-optimal policy; a mismatch in $\\theta$ or a policy that fails the ERAR Bellman equation would refute the claim, and repeating the test with $\\theta$ unfrozen would settle whether the frozen-$\\theta$ experiments changed the fixed point.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Theorem 4, the posterior policy iteration result that EVAL+PPI uses to recover the unregularized greedy average-reward policy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the PPI convergence argument and the observation that iterating the prior policy reaches the $\\beta\\to\\infty$ solution without annealing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the twin/target-network averaging scheme that stabilizes the eigenvector TD updates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Clarifies the centering problem for average-reward value functions, which EVAL's normalized eigenvector solution avoids."}],"review_version":1}