{"id":"389fc8a2-4681-4cf2-b924-8683ffec2c60","arxiv_id":"1908.08379","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"An actor-critic algorithm with a Monte Carlo risk critic is proposed for optimizing reinforcement learning policies under arbitrary, possibly non-coherent risk measures, with a risk function fitted from simulated data.","lead":"This paper proposes a reinforcement learning setup that optimizes policies against general risk measures, not just variance or value-at-risk. It introduces a risk shaping step that fits a risk function from simulation data and a neural risk critic that guides the policy away from risky states.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (10) incorrectly weights the f′ term by ∇P instead of P, so the proposed risk-critic policy update is not the gradient of the stated risk-constrained objective.","rationale":"I read the paper as making a plug-and-play claim: for any differentiable f, Eq. (3) defines a risk measure and Eq. (10) provides a policy gradient update that optimizes the constrained objective. The most load-bearing condition for that claim is that Eq. (10) actually equals ∇E[f(B−J(x))]. It does not: the second term is derived as Pθ ∇f, but in the final expression the probability weight P is replaced by ∇P. The concrete two-outcome computation shows the displayed estimator differs from the true gradient even in the simplest nontrivial case. Since the empirical algorithm explicitly uses Eq. (10), the central architectural claim is unsupported. I do not base this on disagreement with the coherent-risk literature or on the absence of baselines; the mathematical inconsistency is internal to the derivation. The reader's verdict is REJECT, and I agree with that verdict. I mark agreement as partial rather than full because the reader's named weakest assumption is the risk-shaping representation validity, whereas I see the Eq. (10) gradient error as the earlier and more decisive failure. The risk-shaping concern is nevertheless valid and would independently undermine the claim that f can be extracted from data. Both issues would need to be resolved for the paper's central claim to stand.","tokens_in":11325,"tokens_out":3604,"duration_ms":37512,"concrete_test":"Use a two-outcome MDP with B=1 with probability θ and B=0 with probability 1−θ, so J(x)=θ, and take f(u)=u². Then E[f(B−J)] = (1−θ)θ² + θ(1−θ)² = θ(1−θ), whose exact derivative is 1−2θ. Now evaluate the estimator in Eq. (10) for a fixed θ. With P and ∇P for the two trajectories, Eq. (10) evaluates to −1−2θ, not 1−2θ. Recompute the same quantity using the corrected formula E[f(B−J) ∇log P] − ∇J E[f′(B−J)]; the corrected value matches 1−2θ. If the authors intend Eq. (10) to include a 1/P factor on both terms, the displayed formula must be amended accordingly and the implementation in Algorithm 1 checked against this example.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing failure is in the policy-gradient derivation. Starting from Eq. (9), the gradient of R(x)=E[f(B−J(x))] is ∇∑ Pθ(x0:τ) f(B−J(x)) = ∑ [ f(B−J) ∇Pθ(x0:τ) + Pθ(x0:τ) ∇f(B−J(x)) ]. For a fixed trajectory, B does not depend on θ, so ∇f(B−J(x)) = −f′(B−J(x)) ∇J(x). The exact second term is therefore −∇J(x) ∑ Pθ(x0:τ) f′(B−J(x)), i.e., an expectation under the trajectory distribution. Eq. (10), however, writes the second term as ∑ ∇Pθ(x0:τ) [−f′(B−J(x)) ∇J(x)], replacing the probability weight P by ∇P. As printed, the second term is not an expectation and is generally not even properly normalized. This is not a cosmetic typo: Algorithm 1 states that the policy is optimized according to Eqs. (5), (7), and (10), so the implemented update is a biased estimator of the gradient of the risk-constrained objective. The error persists even when f is known exactly and even when the compact reference variants are used, because all variants inherit Eq. (10). Thus the central plug-and-play claim for arbitrary differentiable risk functions is not supported by the derivation. The risk-shaping issue is also real, but this gradient error is more fundamental: it breaks the optimization step even before the question of how f is learned is addressed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a general risk-measure framework for reinforcement learning based on a risk critic that estimates R(x)=E[f(B-J(x))] for any differentiable f, with policy gradient updates under a soft constraint. It introduces 'risk shaping' to learn f from Monte Carlo data and presents an actor-critic architecture (ARCVC) with compact variants. Experiments on a grid world and a gambler's ruin example illustrate different one-sided risk measures and compare sample-based versus network-based penalties.","tokens_in":11708,"tokens_out":6094,"duration_ms":53436,"significance":"If the proposed gradient formula and risk-shaping procedure were correct, the plug-and-play property would be a useful step beyond coherent-risk-only methods. The paper gives a clean decomposition of the objective and a Monte-Carlo-based critic that avoids TD bootstrapping, and the compact architecture is a reasonable engineering idea. However, the central gradient derivation contains a mathematical error and the risk-shaping procedure conflates a conditional expectation with a pointwise function; these issues invalidate the main claims as stated.","major_comments":[{"comment":"The formula for ∇R(x) is mathematically incorrect. From Eq. (9), the gradient is Σ[f(B-J)∇Pθ + Pθ∇f(B-J)], and with ∇f(B-J)=-f'(B-J)∇J, the second term is -∇J Σ Pθ f'. Eq. (10) instead writes both terms under Σ∇Pθ, i.e., Σ∇Pθ[f - f'∇J], omitting the Pθ weight on the f' term. This expression is not the gradient of R(x) and is generally not an expectation under the trajectory distribution. Since Algorithm 1 states that the policy is optimized according to Eqs. (5), (7), and (10), the implemented update is a biased estimator of the gradient of the stated objective. This error is independent of how f is learned and affects all variants.","section":"§3.1, Eq. (10)"},{"comment":"The risk-shaping procedure fits f by plotting R(x)=pk(m) against individual samples of B-J and fitting 1/(1+b(x-c)^2). But Eq. (3) defines R(x) as the expectation of f(B-J), not as a pointwise function of B-J. The relation between a single realization (B-J) and the bankruptcy probability is not f. The paper provides no argument that such a function exists or that this fitting procedure recovers the f appearing in Eq. (3). Without a valid identification argument, the learned f and the resulting risk measure are not meaningful, which undermines the 'risk shaping' contribution.","section":"§5"},{"comment":"The compact objective is self-referential: η(x) is defined as E[B + α(f(B-η(x))-D) g{f(B-η(x))-D}], so η appears on both sides of the equation. The text claims 'it is easy to show' that this converges to J(·) when the constraint is satisfied, but no proof or conditions are given. This is not merely a notation issue; the behavior of the compact architecture depends on the well-posedness of this recursive definition.","section":"§6.1, Eq. (13)"}],"minor_comments":[{"comment":"The phrase 'can be use' should be 'can be used'.","section":"Abstract"},{"comment":"The text says 'any risk function that one can think off', which should be 'think of'.","section":"§1"},{"comment":"The notation ∇Pθ(xt+1|xt) is ambiguous; it should be ∇θ Pθ(xt+1|xt) to clarify that the gradient is with respect to the policy parameters.","section":"§3.1, Proposition 1"},{"comment":"The fitting of 1/(1+b(x-c)^2) is introduced without justification or details on how b and c were estimated; the figure is not described in enough detail to assess the fit.","section":"§5"},{"comment":"The definitions of η(x) and ˆη(x) are inconsistent between the first display and Eq. (13); for example, the first display includes a factor g{E[f(B-ν(x))]-D} while Eq. (13) uses g{f(B-η(x))-D} without the expectation.","section":"§6.1"},{"comment":"The claim that 'it is easy to show' the constraint scaling D, √D, D² for different risk functions is not demonstrated and is not obvious from the definitions.","section":"§7.2"}],"recommendation":"reject","confidential_remarks":"The paper's core contribution is invalidated by the incorrect gradient formula in Eq. (10) and by the questionable identification in the risk-shaping procedure. These are load-bearing errors, not presentational issues. I recommend rejection. A revision that corrects the policy-gradient derivation and provides a principled method for recovering f from data could be reconsidered, but as submitted the main claims are not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a genuinely useful idea buried under a load-bearing error. The risk critic architecture—a Monte-Carlo-trained network that estimates R(x)=E[f(B−J(x))] for arbitrary differentiable f, plus the risk-shaping idea of fitting f from data—is worth thinking about. But Eq. (10) mis-weights the second gradient term: the f′ term is multiplied by ∇P instead of P, so the update Algorithm 1 says to run is not the gradient of the soft-constrained objective. This is not a typo; every variant in the paper inherits it. The risk-shaping section also conflates a conditional expectation (R(x), a ruin probability) with a deterministic function of a single realization, so the fitted bell curve is not shown to be the f from Eq. (3). Those two problems make the central plug-and-play claim unsupported.\n\nWhere the paper deserves credit: it targets a real gap. Coherent risk measures and variance-based criteria badly miss states that are low-variance but high-probability-of-ruin (gambler's ruin is a good example). The Monte-Carlo risk critic is a natural way to handle non-coherent f, and the compact variants (global reference, sample-based penalty) are practical. The literature coverage is fair and the paper is honest about its limitations.\n\nThe soft spots are proportionate: the gradient error is severe; risk shaping is underdeveloped and fitted by hand (b,c on a bell curve); and the experiments compare only the three chosen risk functions to each other, with no baseline like CVaR or variance and no error bars despite 50 repeats.\n\nIf the derivation is corrected—the exact second term is −∇J(x) E[f′(B−J(x))]—the architecture might work, and the risk-shaping idea could be developed into a real procedure with validation. But as printed, the algorithm optimizes a different objective than the one stated.\n\nThis deserves a serious referee: the question is important, the approach is plausible, and the error is pointed enough that a careful revision could fix it. I would not cite it in its current form. Take it to the next reading group if you want a concrete case study in policy-gradient derivations; otherwise wait for a revised version.","headline":"A genuinely useful risk-critic idea, undercut by a load-bearing gradient error in Eq. (10) and a muddled risk-shaping step; both need fixing before the plug-and-play claim can stand.","tokens_in":12170,"tokens_out":3978,"would_cite":false,"duration_ms":39024,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","90C40"],"pacs":[],"model":"deepseek-v4-flash","headline":"A single risk critic can optimize policies under any differentiable risk measure, the paper argues.","keywords":["risk measures","reinforcement learning","policy gradient","risk shaping","risk critic","non-coherent risk","Monte Carlo estimation","actor-critic"],"falsifier":"Fit $f$ on the gambler's ruin problem for a fixed look-ahead $k$, then compute $\\mathbb{E}[f(B-J(x))]$ under the same process and compare the result with the true bankruptcy probability $p_k(m)$; if they diverge materially, Eq. (3) does not represent the bankruptcy risk.","tokens_in":11134,"feed_emoji":"🛡️","tokens_out":5215,"duration_ms":48772,"temperature":0.7,"pith_summary":"This paper tries to establish that a reinforcement learning agent can optimize a policy under a general, user-specified risk measure without designing a new algorithm for each one. It defines the risk at a state as $R(x)=\\mathbb{E}[f(B-J(x))]$, where $B$ is the discounted reward-to-go and $f$ is any differentiable shaping function, and derives a policy-gradient update that works for every such $f$. The paper also introduces risk shaping: fitting $f$ from Monte Carlo samples so the risk measure reflects the problem's actual failure modes, which need not be coherent or convex. If the representation is sound, practitioners could choose or learn an arbitrary risk shape and reuse a single actor-critic architecture.","feed_headline":"One risk critic optimizes any differentiable risk measure","feed_subtitle":"Fit the risk shape from data, plug it into policy gradient, and enforce risk constraints across reinforcement learning.","key_machinery":"The load-bearing object is the risk functional $R(x)=\\mathbb{E}[f(B-J(x))]$ together with its policy gradient, given in the paper's Eq. (10), which augments the standard likelihood-ratio gradient with the term $f'(B-J(x))\\nabla J(x)$. Around this, the paper builds the ARCVC architecture: an actor for the policy, a value network estimating $J$, and a risk network estimating $R$, trained on a finite-time buffer of Monte Carlo reward-to-go samples through the losses in Eqs. (11) and (12). Risk shaping fits $f$ to empirical pairs $(B-J, R(x))$ obtained from simulation, allowing $f$ to be non-convex and non-coherent.","core_discovery":"The paper's central claim is that a general risk measure can be written as $R(x)=\\mathbb{E}[f(B-J(x))]$, where $B$ is the reward-to-go, $J(x)$ is the expected reward-to-go from state $x$, and $f$ is any differentiable function such as the square, absolute value, or square root. For this family, the paper derives an explicit policy-gradient update that includes a correction term involving $f'(B-J(x))\\nabla J(x)$, because the reference $J$ itself depends on the policy. It then shows how to extract $f$ from data through risk shaping, using a gambler's ruin example where the natural bankruptcy-risk probability is non-convex and where variance is a poor risk indicator. The proposed actor-risk-critic-value-critic architecture estimates the risk from finite-time Monte Carlo buffers and uses it in a soft constraint, with experiments showing that non-coherent one-sided risk functions can reduce risk violations while sometimes lowering success rate.","pith_inferences":["If the Eq. (3) representation holds beyond the examples, the same critic could be trained against distributional estimates of $B-J$ and applied to continuous-control safety, where many risk shapes are currently handled one at a time.","The risk-shaping step as described treats $f$ as fixed once fitted; interleaving fitting and policy optimization, which the paper flags as future work, would turn risk shaping into an online meta-critic.","The closeness of global and state references at high $\\gamma$ suggests long-horizon tasks may be the easiest targets for the compact architecture, because the reward-to-go distribution becomes dominated by the mean over states."],"forward_implications":["Any differentiable $f$ can be used in the same policy-gradient machinery, so variance, one-sided penalties, and quantile-like shapes become special cases of one update rule.","A risk constraint enforced during training acts as safe exploration; in the experiments the one-sided square-root risk cuts risk violations while lowering success rate.","Replacing the state-wise reference $J(x)$ with a global scalar $\\bar{J}$ costs little accuracy as the discount factor $\\gamma$ approaches 1, enabling a cheaper two-network or one-network agent.","Replacing the risk network with a single sample in the penalty function saves memory and runtime at the price of worse accumulated reward and higher violation rate.","The extracted $f$ in the gambler's ruin example is non-convex, so restricting to coherent risk measures would exclude a natural bankruptcy-risk shape."],"supporting_citations":[{"why":"Supplies the variance-related policy-gradient method that this paper generalizes to arbitrary differentiable risk functions.","marker":"[Tamar et al., 2012]"},{"why":"Provides the likelihood-ratio policy-gradient estimator used in the derivation of Proposition 1.","marker":"[Baxter and Bartlett, 2001]"},{"why":"Reward shaping inspires the paper's risk-shaping procedure for extracting $f$ from data.","marker":"[Ng et al., 1999]"},{"why":"Presents the coherent-risk policy-gradient baseline that the paper claims to surpass in computational simplicity and generality.","marker":"[Tamar et al., 2015a]"},{"why":"Supplies the actor-critic and policy-gradient framework and the grid-world setting used in the experiments.","marker":"[Sutton and Barto, 2018]"},{"why":"Shows that closed-form risk forms such as variance exist only in limited cases, motivating the Monte Carlo estimation approach.","marker":"[Sobel, 1982]"}],"fun_headline_variants":["Risk critic learns any differentiable risk measure","Optimize RL under non-coherent risk, not just variance","Policy gradient with correction for risk-shaped rewards","Learn risk from data, then optimize with a risk critic","General risk measures in RL via neural risk critic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The risk measure of interest can be reproduced by fitting a single function $f$ in $R(x)=\\mathbb{E}[f(B-J(x))]$ from pointwise samples of $B-J$ versus $R(x)$; the paper does not prove that this representation is valid for a given risk measure.","fun_headline_variants_meta":{"raw":{"variants":["Risk critic learns any differentiable risk measure","Optimize RL under non-coherent risk, not just variance","Policy gradient with correction for risk-shaped rewards","Learn risk from data, then optimize with a risk critic","General risk measures in RL via neural risk critic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1181,"prompt_tokens":838,"completion_tokens":343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":454,"tokens_out":343,"duration_ms":4031,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:43:06.838960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit $f$ on the gambler's ruin problem for a fixed look-ahead $k$, then compute $\\mathbb{E}[f(B-J(x))]$ under the same process and compare the result with the true bankruptcy probability $p_k(m)$; if they diverge materially, Eq. (3) does not represent the bankruptcy risk.","supporting_citations":[{"cited_title":"Policy gradients with variance related risk criteria","cited_arxiv_id":null,"evidence_quote":"Supplies the variance-related policy-gradient method that this paper generalizes to arbitrary differentiable risk functions."},{"cited_title":"Infinite-horizon policy-gradient estimation","cited_arxiv_id":null,"evidence_quote":"Provides the likelihood-ratio policy-gradient estimator used in the derivation of Proposition 1."},{"cited_title":"The variance of discounted markov decision processes","cited_arxiv_id":null,"evidence_quote":"Shows that closed-form risk forms such as variance exist only in limited cases, motivating the Monte Carlo estimation approach."}],"review_version":1}