{"id":"d344c243-f598-43d7-a048-58d8759bd99f","arxiv_id":"2507.03900","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A bi-level actor-critic framework optimizes static spectral risk measures in online and offline RL, with tabular convergence guarantees and experiments on trading, portfolio, HIV, and MuJoCo tasks.","lead":"This paper introduces actor-critic algorithms that optimize static spectral risk measures (a family including CVaR) for reinforcement learning, in both online and offline settings. It is a step toward risk-sensitive agents that honor a fixed worst-case budget over whole episodes rather than at every decision step.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.3 proves monotonic improvement for exact alternating maximization with the exact h_phi,G, but Algorithm 1 uses the approximate tilde h and finite inner steps, so the convergence guarantee does not cover the implemented method.","rationale":"The reader's weakest_assumption identifies the same core issue: Theorem 4.3 is proved for exact alternating maximization with the exact risk-function maximizer, while the algorithm uses the approximate tilde h and a finite number of inner policy-gradient steps. I agree that this is load-bearing because it separates the convergence theorem from the method whose convergence is claimed. I sharpened the point in two ways: the equality in Algorithm 1 Step 1 between argmax_h J(pi_k,h) and tilde h is not an equality, and the compactness assumption on Theta_0 is not merely unproven but is violated by the softmax parameterization's per-state logit shift invariance. I did not elevate the Lemma 1 discount-factor inconsistency to the headline because it appears to be a fixable proof typo: the correct Bellman relation in the extended MDP is Q_h = gamma E[V_h(next)], and the telescoping can be rewritten with c_t=gamma^t; the convergence conclusion may survive with adjusted constants. The approximation/exactness gap is structural and directly affects whether the central claim applies to the submitted algorithms. The verdict should remain CONDITIONAL: the framework is plausible and the experiments are substantial, but the convergence guarantee must be restated and proved for the approximate implementation, or the manuscript must explicitly present Theorem 4.3 as a guarantee for an idealized limiting version and add an approximation-error analysis for the practical one.","tokens_in":32277,"tokens_out":11551,"duration_ms":143004,"concrete_test":"Independently derive the monotonicity step for the actual update h_{k+1}=tilde h_{phi,G_pi_k}: bound the deficit J(pi_k)-E[tilde h(G_pi_k)] in terms of sup_z |tilde h(z)-h_phi,G(z)| and the inner-loop suboptimality. If no bound can be established, run Algorithm 2 on a small tabular MDP (e.g., 5 states, N=10 quantiles, T_inner=10 gradient steps) and record J(pi_{k+1})-J(pi_k) over T_outer iterations; one observed decrease falsifies the monotonic-improvement theorem for the implemented method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central convergence claim is that Algorithm 1 directly optimizes J(pi)=SRM_phi(G_pi). Algorithm 1 Step 1 sets h_{k+1}=argmax_h J(pi_k,h)=tilde h_{phi,G_pi_k}, and Algorithms 2-4 update h as tilde h and take only T_inner policy-gradient steps. The proof of Theorem 4.3, however, uses exactness of both blocks: it asserts that h_{k+1} 'is constructed using the quantiles of G_pi_k to maximize E[h(G_pi_k)]' and that pi_{k+1} is the maximizing policy for h_{k+1}. This is not what the algorithm computes. Proposition 4.1 explicitly derives tilde h as an approximation of the true maximizer h_phi,G, using a midpoint quadrature, and the conjugate term int hat h(phi(u))du, which vanishes only for the exact maximizer, is dropped in the Q-value computation. No bound is given on sup_z |tilde h(z)-h_phi,G(z)|, nor on the inner-loop suboptimality after T_inner steps, so the monotonic chain J(pi_{k+1}) >= E[tilde h(G_pi_{k+1})] >= E[tilde h(G_pi_k)] = J(pi_k) is unjustified for the actual algorithm. Separately, the compactness assumption on Theta_0 is not verified; for the softmax parameterization used in Theorem 4.2, adding a per-state constant to the logits leaves the policy and J unchanged, so the sublevel set is noncompact. Thus the advertised convergence guarantee for the proposed actor-critic framework is unsupported as stated, even though the underlying bi-level idea may be salvageable with explicit error control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes actor-critic algorithms (AC-SRM, OAC-SRM, TD3-SRM, TD3BC-SRM) for directly optimizing static spectral risk measures (SRM) of the return distribution in both online and offline reinforcement learning. The method is based on a bi-level reformulation of the SRM via its supremum representation, alternating an inner policy optimization for a fixed risk function h and an outer update of h using quantiles of the current return distribution. The authors prove convergence of the inner policy optimization in the finite tabular setting (Theorem 4.2), monotonic improvement and stationarity of the overall procedure (Theorem 4.3), and derive a risk-sensitive advantage-weighted update for the offline setting (Theorem 4.4). They report empirical results on trading, portfolio allocation, HIV treatment, and stochastic MuJoCo environments, comparing against risk-neutral and iterative-risk baselines.","tokens_in":32585,"tokens_out":7676,"duration_ms":98240,"significance":"If the convergence claims were fully supported, this would be a useful contribution: static SRM objectives are known to be poorly served by per-step iterative risk measures, and an actor-critic method that directly optimizes a static SRM with guarantees would fill a real gap in risk-sensitive RL, especially for offline policy tailoring. The supremum-representation reformulation is conceptually clean, the empirical study is broad and includes a compelling portfolio-allocation demonstration, and the authors provide code and detailed hyperparameters. However, the main theoretical results currently rest on a performance-difference lemma whose proof has a dimensional inconsistency, and the implemented algorithms use an approximate risk function and finite inner-loop updates that are not covered by the stated monotonicity theorem. These issues are load-bearing for the paper's central claim, so the contribution is not yet established as written.","major_comments":[{"comment":"","section":"Appendix C, Lemma 1 (Eq. 16)"},{"comment":"","section":"Theorem 4.3 and Algorithm 1 (also Algorithms 2–4)"},{"comment":"","section":"Appendix D, compactness assumption in Theorem 4.3"},{"comment":"","section":"Equation (9) and Lemma 1 interplay"}],"minor_comments":[{"comment":"","section":"Equation (15)"},{"comment":"","section":"Algorithms 2–4"},{"comment":"","section":"Section 3.3"},{"comment":"","section":"Section 5.5 / Figure 1"},{"comment":"","section":"Appendix C, Eq. (17)"}],"recommendation":"major_revision","confidential_remarks":"The central convergence theorem is not supported by the current proofs: the performance-difference lemma in Appendix C appears to have a genuine dimensional error, not merely a typo, and the implemented algorithms do not match the exact-update assumptions of Theorem 4.3. The compactness assumption also fails for the softmax parameterization used in Theorem 4.2. The bi-level idea and empirical work are promising, but the theoretical claims need substantial repair. I would suggest the editor obtain a second opinion on the correctness of Lemma 1 before committing to a revise-and-resubmit decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on 2507.03900. The genuinely new thing is the actor-critic and offline instantiations of static spectral risk measures—AC-SRM, OAC-SRM, TD3-SRM, TD3BC-SRM—extending the authors' earlier value-based work. That's a natural and potentially useful contribution, and the derivation of the closed-form h via the supremum representation is clean. The extended-state construction (s, c) and the Q-value normalization by c are sensible. The experimental scope is broad: trading, portfolio allocation, HIV, stochastic MuJoCo. The code appears to be public. The circularity worry you flagged does not land: the h-update is a legitimate variational step, not a fitted target. All that is real.\n\nThe soft spots, in order of size. First, Theorem 4.3 as stated does not cover Algorithm 1. The proof assumes exact alternating maximization: exact h_{φ,G} and exact inner-loop policy optimization. The algorithm uses the approximate piecewise-linear tilde h from Proposition 4.1 and runs a finite number T_inner of policy-gradient steps. No bound is given on the approximation error or the inner-loop suboptimality, so the monotonic improvement chain is unjustified for the implemented method. This is load-bearing. It is not necessarily fatal—the bi-level idea could be rescued with explicit error control or by proving convergence to a neighborhood—but as written the theorem overclaims. Second, the compactness of Theta_bar is asserted as 'standard,' but for softmax policies the sublevel set is noncompact due to translation invariance; adding a constant to all logits leaves the policy unchanged. The proof needs to handle the quotient or add regularization. Third, Lemma 1's proof has a discount-factor slip: step (a) writes c_t V_{t+1} where it should be c_{t+1} V_{t+1}. This looks fixable—the equality holds after the correction—but the displayed step is wrong. Fourth, the abstract says 'consistently outperform existing risk-sensitive methods,' but the paper's own tables show risk-neutral baselines beating some of the new methods on several tasks (e.g., Hopper and HalfCheetah offline for mean return). The discussion section is more honest, but the abstract overstates.\n\nOverall: the paper deserves peer review. It is a legitimate extension with plausible algorithms and some useful theory, but the main convergence guarantee needs to be either repaired or significantly qualified, and the empirical claims need to be toned down. I would send it out and expect major revisions.","headline":"A real algorithmic extension of static-SRM RL to actor-critic and offline settings, but the headline convergence theorem does not cover the implemented approximate algorithm and the abstract oversells the experiments.","tokens_in":33197,"tokens_out":5497,"would_cite":true,"duration_ms":56767,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C40","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual-loop actor-critic optimizes static spectral risk measures directly, in both online and offline reinforcement learning, with convergence guarantees in finite MDPs.","keywords":["risk-sensitive reinforcement learning","spectral risk measures","static risk measures","actor-critic","offline reinforcement learning","distributional reinforcement learning","CVaR","convergence guarantees"],"falsifier":"Run AC-SRM or TD3-SRM on a small finite MDP where $J(\\pi) = \\mathrm{SRM}_\\varphi(G^\\pi)$ can be computed exactly by enumerating all trajectories, and record $J(\\pi_k)$ at each outer iteration. A single decrease $J(\\pi_{k+1}) < J(\\pi_k)$, or a computed gradient sequence whose norm does not approach zero, would falsify Theorem 4.3 as stated. The toy setting needs only two states, two actions, a small horizon, and a fixed spectrum such as CVaR with $\\alpha = 0.2$.","tokens_in":31936,"feed_emoji":"📉","tokens_out":7564,"duration_ms":77974,"temperature":0.7,"pith_summary":"The paper addresses a known flaw in risk-sensitive reinforcement learning: applying a risk measure at every time step, as distributional RL usually does, does not optimize any fixed risk measure of the whole trajectory. It proposes instead to directly maximize a static Spectral Risk Measure (SRM) of the total return, using a bi-level actor-critic method in which an outer loop updates a concave risk function based on the return distribution and an inner loop maximizes the policy under that function. The framework is instantiated for stochastic and deterministic policies in both online and offline settings, named AC-SRM, OAC-SRM, TD3-SRM, and TD3BC-SRM. The paper proves convergence of the inner policy search and monotonic improvement of the SRM objective in the finite state-action tabular setting, and reports strong empirical performance against iterative-risk baselines in finance, healthcare, and robotics benchmarks.","feed_headline":"Dual-loop actor-critic optimizes static spectral risk measures","feed_subtitle":"Convergence holds for finite MDPs, and one offline dataset yields policies for every risk profile.","key_machinery":"The load-bearing object is the supremum representation of a Spectral Risk Measure: $\\mathrm{SRM}_\\varphi(Z) = \\sup_{h \\in H}\\left(\\mathbb{E}[h(Z)] + \\int_0^1 \\hat{h}(\\varphi(u))\\,du\\right)$, with the supremum attained at $h_{\\varphi,Z}(z) = \\int_0^1 \\left(F_Z^{-1}(\\alpha) + \\frac{1}{\\alpha}(z - F_Z^{-1}(\\alpha))^-\\right)\\mu(d\\alpha)$. Proposition 4.1 approximates this $h$ by a piecewise-linear function built from $N$ quantiles with weights $w_i = \\hat{\\tau}_i(\\varphi(\\tau_{i-1}) - \\varphi(\\tau_i))$, so the risk preference is re-expressed as a deterministic, action-independent function of the return. This $h$ is then used to define $Q$-values on an extended state space (accumulated reward $s$ and discount $c$), which is what lets the same global risk spectrum act on returns from different time steps. The risk-sensitive Performance Difference Lemma converts the advantage function into policy improvement and is the engine behind both the inner convergence theorem and the offline advantage-weighted update.","core_discovery":"The central claim is that $J(\\pi) = \\mathrm{SRM}_\\varphi(G^\\pi)$, the spectral risk measure of the return distribution from the initial state, can be optimized by alternating two steps instead of embedding the risk measure into every Bellman update. For a fixed concave function $h$, the inner loop maximizes $J(\\pi, h) = \\mathbb{E}[h(G^\\pi)] + \\int_0^1 \\hat{h}(\\varphi(u))\\,du$ via a risk-sensitive policy gradient with $Q_h^\\pi(\\bar{x}, a) = \\mathbb{E}[h(s + c G^\\pi(\\bar{x}, a))]/c$; the outer loop sets $h_{k+1} = \\tilde{h}_{\\varphi, G^{\\pi_k}}$, a piecewise-linear quantile approximation of the exact maximizer from Proposition 4.1. Theorem 4.2 establishes that the inner natural-policy-gradient updates converge to the optimal softmax policy for fixed $h$, and Theorem 4.3 establishes that the alternating scheme monotonically improves $J(\\pi)$, converges, and has $\\liminf_k \\|\\nabla_\\theta J(\\pi_{\\theta_k})\\| = 0$. The authors also derive an advantage-weighted update for the offline constraint setting. If correct, this gives an actor-critic family that optimizes static SRM directly, covering CVaR and Mean-CVaR as special cases, and avoids the time-inconsistency of iterative risk measures.","pith_inferences":["Our inference: the framework should extend to any coherent distortion risk measure with an absolutely continuous distortion function, since the SRM representation covers those; experiments with Wang or proportional-hazard spectra would test this directly.","Our inference: the time-inconsistency advantage should grow with horizon length and tail severity, so longer-horizon stochastic MuJoCo tasks or rare-disaster reward models should show larger margins over iterative-risk baselines.","Our inference: the paper's observation about entropy-based critics suggests a decoupled risk-sensitive SAC where entropy is not risk-adjusted; this modification is not implemented in the paper but is a natural testable extension.","Our inference: if the approximate $\\tilde{h}$ is replaced by an exact or better-quantized $h$ in the outer loop, Theorem 4.3's monotonicity could be brought closer to the implemented algorithm; comparing the two on a finite MDP would quantify the approximation gap."],"forward_implications":["Policies can be matched to a user's true risk preference: the risk measure is applied once to the whole return, not step by step, so CVaR- or Mean-CVaR-optimizing policies are no longer distorted by local risk decisions.","For offline RL, one fixed dataset yields a whole family of risk-tuned policies: changing the spectrum parameter (e.g., $\\omega$ in Mean-CVaR) gives different risk-return profiles without further environment interaction.","The same bi-level scheme works for stochastic actors (AC-SRM, OAC-SRM) and deterministic actors (TD3-SRM, TD3BC-SRM), so discrete and continuous control can share the static-SRM objective.","In the finite tabular case, the inner policy search converges to the optimal softmax policy for the fixed $h$, and the overall loop monotonically improves the static SRM objective.","Iterative risk baselines such as DSAC-iCVaR and CODAC-iMC are consistently outperformed in the reported trading, portfolio, HIV, and MuJoCo experiments."],"supporting_citations":[{"why":"supplies the closed-form quantile solution for the risk function h and the distributional SRM approach this work extends to actor-critic.","marker":"[Moghimi and Ku, 2025]"},{"why":"defines Spectral Risk Measures and the risk-spectrum representation used throughout the paper.","marker":"[Acerbi, 2002]"},{"why":"provides the supremum and conjugate representation of SRM that motivates the bi-level split.","marker":"[Pichler, 2015]"},{"why":"introduces the extended state space and the decomposition into a risk function and a policy optimization.","marker":"[Bäuerle and Glauner, 2021]"},{"why":"natural policy gradient scheme used in Theorem 4.2's inner-loop convergence proof.","marker":"[Kakade, 2001]"},{"why":"supplies the finite-MDP policy-gradient convergence framework and Fisher-information cancellation lemma used in Theorem 4.2.","marker":"[Agarwal et al., 2021]"},{"why":"gives the advantage-weighted policy update under a KL constraint adapted into Theorem 4.4 for offline risk-sensitive learning.","marker":"Peng et al. [2019]"},{"why":"block-coordinate-descent convergence result used to structure the alternating-update proof of Theorem 4.3.","marker":"[Tseng, 2001]"},{"why":"QR-DQN quantile regression loss used to implement the distributional critic.","marker":"[Dabney et al., 2018b]"},{"why":"TD3 mechanisms (clipped double Q, delayed updates, target smoothing) reused in TD3-SRM.","marker":"[Fujimoto et al., 2018]"}],"fun_headline_variants":["Alternating actor-critic optimizes static spectral risk","Dual-loop RL for spectral risk measures covering CVaR","Risk-sensitive actor-critic with static SRM guarantees","Online and offline RL via static spectral objectives","Actor-critic convergence for spectral risk in MDPs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The convergence proofs assume the exact outer maximizer $h_{\\varphi, G^{\\pi_k}}$ and a compact parameter sublevel set, while the implemented algorithm uses a piecewise-linear approximation $\\tilde{h}$ and finitely many inner policy updates; the monotonic-improvement guarantee does not strictly follow for the implemented approximation unless its error is controlled.","fun_headline_variants_meta":{"raw":{"variants":["Alternating actor-critic optimizes static spectral risk","Dual-loop RL for spectral risk measures covering CVaR","Risk-sensitive actor-critic with static SRM guarantees","Online and offline RL via static spectral objectives","Actor-critic convergence for spectral risk in MDPs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1456,"prompt_tokens":1025,"completion_tokens":431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":354}},"tokens_in":641,"tokens_out":431,"duration_ms":6072,"temperature":1.0,"reasoning_tokens":354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:01:32.427387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AC-SRM or TD3-SRM on a small finite MDP where $J(\\pi) = \\mathrm{SRM}_\\varphi(G^\\pi)$ can be computed exactly by enumerating all trajectories, and record $J(\\pi_k)$ at each outer iteration. A single decrease $J(\\pi_{k+1}) < J(\\pi_k)$, or a computed gradient sequence whose norm does not approach zero, would falsify Theorem 4.3 as stated. The toy setting needs only two states, two actions, a small horizon, and a fixed spectrum such as CVaR with $\\alpha = 0.2$.","supporting_citations":[{"cited_title":"Beyond CVaR : Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"supplies the closed-form quantile solution for the risk function h and the distributional SRM approach this work extends to actor-critic."},{"cited_title":"Minimizing spectral risk measures applied to Markov decision processes","cited_arxiv_id":null,"evidence_quote":"introduces the extended state space and the decomposition into a risk function and a policy optimization."},{"cited_title":"A Natural Policy Gradient","cited_arxiv_id":null,"evidence_quote":"natural policy gradient scheme used in Theorem 4.2's inner-loop convergence proof."},{"cited_title":"Kakade, Jason D","cited_arxiv_id":null,"evidence_quote":"supplies the finite-MDP policy-gradient convergence framework and Fisher-information cancellation lemma used in Theorem 4.2."},{"cited_title":"Advantage- Weighted Regression : Simple and Scalable Off-Policy Reinforcement Learning , 2019","cited_arxiv_id":null,"evidence_quote":"gives the advantage-weighted policy update under a KL constraint adapted into Theorem 4.4 for offline risk-sensitive learning."},{"cited_title":"Addressing Function Approximation Error in Actor-Critic Methods","cited_arxiv_id":null,"evidence_quote":"TD3 mechanisms (clipped double Q, delayed updates, target smoothing) reused in TD3-SRM."}],"review_version":1}