REVIEW 3 major objections 5 minor 36 references
A PPO agent that jointly chases return and reliability can rebalance portfolios under transaction costs and tail-risk measures as well as classical multi-objective optimizers across market regimes.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 01:53 UTC pith:P3CFOFEO
load-bearing objection Solid integration of reliability constraints + transaction costs into PPO portfolio RL, but the EVaR table is broken and the three-risk claim does not fully hold. the 3 major comments →
Deep Reinforcement Learning for Reliability Based Bi-Objective Portfolio Optimization
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
MORP-DRL—a PPO actor–critic trained on reliability-shaped rewards and GARCH–EVT–t-copula scenarios—produces sequential portfolio policies that match or approach NSGA-II on return, volatility, Sharpe, and the chosen risk measure while satisfying transaction-cost and bound constraints across three market regimes and remaining usable on the FTSE 100 universe.
What carries the argument
The reliability-shaped PPO reward: net-return and risk-improvement terms plus a piecewise penalty Ψ(ρ, β) that rewards estimated return and risk reliability above prescribed thresholds β1 = β2 = 0.65, with scenarios generated by GARCH(1,1), EVT, t-copula, and quasi-Monte Carlo.
Load-bearing premise
That reliability probabilities estimated from the same scenario generator used in training are faithful enough proxies for true out-of-sample constraint satisfaction that shaping the reward with them produces genuinely reliable policies rather than scenario-overfit ones.
What would settle it
Hold out a later market window never used in scenario generation or training; recompute empirical return and risk reliability of the frozen PPO policy on that window and check whether both still clear the 0.65 thresholds while risk–return metrics remain competitive with NSGA-II.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MORP-DRL, a PPO-based multi-objective reliability portfolio framework that jointly optimizes expected return and downside risk under variance, CVaR, and EVaR, with proportional transaction costs and probabilistic reliability constraints. Uncertainty is modeled via GARCH(1,1), EVT (GPD tails), a t-copula, and QMC scenarios; reliability estimates enter a shaped reward. The method is tested on ten global equity indices over pre-COVID, COVID, and post-COVID regimes and scaled to FTSE 100 constituents, with NSGA-II and equal-weight benchmarks. The authors claim competitive risk–return performance, better downside control in stress periods, and high-dimensional scalability.
Significance. If the three-risk-measure results hold under a consistent evaluation protocol, the paper would be a useful empirical contribution at the intersection of reliability-based design optimization and DRL portfolio management: it couples sequential rebalancing, transaction costs, and coherent tail-risk objectives with scenario-based reliability constraints, and provides a concrete FTSE 100 scaling path via time-decomposed QMC. The variance and CVaR legs (Tables 1–2) already show both optimizers beating equal weight with plausible magnitudes, and the allocation/concentration appendices usefully document diversification differences between PPO and NSGA-II. The EVaR leg is currently the load-bearing weak point for the multi-measure claim.
major comments (3)
- [§7 Table 3; also §3.1, Model C Eq. (8), §5.1] Table 3 (Model C, EVaR): reported EVaR levels are not on a common scale. Base and NSGA-II EVaR sit near 4.75 across all three regimes, while PPO reports 0.0013 (Pre), 0.0073 (COVID), and 0.0124 (Post)—orders of magnitude smaller—yet Sharpe/return are still presented as comparable. Section 3.1 defines EVaR via the standard exponential moment form, and Models B/C and §5.1 use the same risk object for reward and reliability, but the manuscript never states units, annualization, a different estimator, or a different evaluation protocol for PPO vs NSGA-II. Without a reconciled definition and recomputed Table 3 (and the corresponding Pareto frontiers in Figs. 6–7), the claim that MORP-DRL is competitive under all three risk measures, and that it reduces downside risk under EVaR in stress periods, is not supported by the reported numbers.
- [§5.1 reward/reliability; §6.2; §7] Reliability evaluation is largely in-sample relative to the training scenario generator. §5.1 estimates ρ_ret and ρ_risk from the same GARCH–EVT–t-copula–QMC scenarios used to train PPO, then shapes the reward with Ψ(ρ,β) at β1=β2=0.65 (§6.2). Tables 1–4 report return/volatility/Sharpe/risk but do not report realized reliability frequencies on held-out paths, alternative copulas, or bootstrap stress tests. For a reliability-based claim, the paper should show that the learned policies actually meet the prescribed β levels out of the training scenario measure; otherwise the reliability component may be scenario-overfit rather than a genuine feasibility guarantee.
- [§4 Models A–C; §5.1 Eq. (1); §7 Figs. 2–7] The bi-objective / multi-objective framing is only partially realized in the DRL path. Models A–C are written as bi-objective programs (max return, min risk) with reliability constraints, and NSGA-II produces Pareto frontiers, but PPO optimizes a single scalar reward r_t = ΔS_t + ΔR_t + R_rel_t (Eq. 1) with fixed hand-tuned scales in Ψ. The paper should clarify how the reported PPO "Pareto frontiers" (Figs. 3, 5, 7) are generated (e.g., reward-weight sweeps, threshold sweeps, or post-hoc filtering) and whether they are comparable to NSGA-II’s nondominated sets under the same constraint set. Without that protocol, claims of multi-objective competitiveness rest on single-policy snapshots rather than a controlled frontier comparison.
minor comments (5)
- [Title; Abstract; §1] Title and abstract say "bi-objective" while the framework name is multi-objective reliability based (MORP-DRL) and three risk measures are treated as alternative models; align terminology throughout.
- [§5.1 Eq. (1)] Eq. (1) writes S_t = R_t / R_t (typo); the intended risk-adjusted score should be stated explicitly (e.g., return/risk or Sharpe-like).
- [Algorithm 1; §5.2] Action sampling is described as a_t ~ Dirichlet(π_θ(s_t)) then projected onto bounds; specify how the policy network parameterizes the Dirichlet and how bound projection interacts with the PPO probability ratio.
- [Tables 1–3; §7] CPU times in Tables 1–3 show PPO ~10–15× slower than NSGA-II; a short note on wall-clock fairness (episode count, scenario budget, hardware) would help interpret the efficiency comparison in §7.
- [§1; References] Several repeated Markowitz citations and minor formatting issues (e.g., "capital asset Pricing model", missing spaces in headings) should be cleaned for production.
Circularity Check
Empirical RL portfolio method; reward-shaped reliability is design, not a by-construction prediction of the reported metrics.
specific steps
-
other
[§5.1 Reward Function and Reliability Constraints; Algorithm 1 lines 6, 15–16]
"Generate market scenarios using GARCH(1,1), EVT, t-copula, and QMC ... Estimate reliability probabilities: ρ_ret_t , ρ_risk_t ... Compute reward r_t using Eq. (1) ... R_rel_t = Ψ(ρ_ret_t , β1) + Ψ(ρ_risk_t , β2)"
Reliability probabilities that shape the PPO reward are estimated from the same GARCH–EVT–t-copula–QMC scenarios used to train the agent. This couples the training objective to the scenario model, so high in-sample ρ does not independently prove out-of-sample reliability. It is environment design, not a mathematical reduction of the paper’s reported historical return/Sharpe claims to their inputs; evaluation remains on separate historical regimes.
full rationale
MORP-DRL is a methods paper: it formulates bi-objective reliability-constrained portfolio models (variance/CVaR/EVaR with transaction costs), generates scenarios via GARCH–EVT–t-copula–QMC, trains PPO with a hand-designed reward that includes reliability penalties Ψ(ρ,β), and evaluates annualized return, volatility, Sharpe, and risk on historical pre/COVID/post regimes against equal-weight and NSGA-II. That chain is not a first-principles derivation that redefines its target as an input. The reliability terms in the reward use the same scenario generator used for training, which is standard RL environment design and can cause scenario-overfit, but evaluation metrics (Tables 1–4) are computed on realized historical series, not tautologically equal to the fitted reward components. Self-citations to prior reliability portfolio work ([32] Sengupta et al.) are used as baselines that the paper extends with transaction costs and DRL; they are not uniqueness theorems that force the present claims. No equation reduces a reported prediction to a fitted parameter by construction. Numerical inconsistencies in Table 3 EVaR scales (NSGA-II ~4.75 vs PPO ~0.001–0.012) are a correctness/reporting issue, not circularity of the derivation. Score 1 for mild shared-scenario design only.
Axiom & Free-Parameter Ledger
free parameters (8)
- reliability thresholds β1, β2 =
0.65
- portfolio weight bounds wi,min, wi,max =
0.03 ≤ wi ≤ 0.35
- proportional transaction cost ki =
2 bps
- reliability reward scales in Ψ =
+30 / −100 (and ×100)
- PPO training hyperparameters =
AdamW 3e-4, γ=0.99, ε=0.2, 1000 episodes
- EVT exceedance threshold =
95th percentile
- GARCH(1,1) and t-copula parameters =
asset- and period-specific (not tabulated)
- target thresholds r*_p, σ²*_p, CVaR*, EVaR* =
not fully specified numerically in §6
axioms (5)
- domain assumption Asset returns are adequately described by GARCH(1,1) conditional volatility, GPD tails above a high threshold, and a Student-t copula dependence structure, with QMC scenarios representing future uncertainty.
- domain assumption Portfolio choice is an MDP with continuous weights on the simplex and unknown transition kernel learnable from historical/simulated paths.
- domain assumption CVaR and EVaR are coherent risk measures suitable as bi-objective risk legs alongside expected return under reliability constraints.
- standard math Clipped PPO surrogate with Dirichlet-sampled actions and projection onto weight bounds yields stable continuous portfolio policies.
- ad hoc to paper Pre-COVID / COVID / Post-COVID calendar splits (2018–2019 / 2020–2021 / 2022–2023) are meaningful distinct market regimes for evaluating adaptation.
invented entities (1)
-
MORP-DRL reward (ΔS + ΔR + R_rel with Ψ reliability shaping)
no independent evidence
read the original abstract
Portfolio optimization under uncertainty is inherently a multi-objective decision problem involving complex interactions among return, risk, market dynamics, and practical investment constraints. Existing reliability based portfolio optimization approaches primarily rely on static optimization frameworks and often fail to capture sequential decision making, tail risk, and market frictions such as transaction costs. To address these limitations, we propose a deep reinforcement learning framework for multi-objective reliability based portfolio optimization (MORP-DRL). The proposed framework jointly optimizes expected return and downside risk using three complementary risk measures: variance, Conditional Value-at-Risk (CVaR), and Entropic Value-at-Risk (EVaR). To model uncertainty and heavy-tailed market behavior, asset returns are represented using GARCH(1,1), Extreme Value Theory, and a t-copula dependence structure, while realistic scenarios are generated through quasi-Monte Carlo simulation. A Proximal Policy Optimization (PPO) based strategy is developed under practical constraints including transaction costs and portfolio bounds, and is benchmarked against NSGA-II. Experiments on ten global equity indices across pre-COVID, COVID, and post-COVID market regimes demonstrate that MORP-DRL achieves competitive risk-return performance, reduced downside risk during periods of market stress, and scalability to high-dimensional portfolio settings.
Figures
Reference graph
Works this paper leans on
-
[2]
Portfolio selection.Handbook of finance, 2:3–13, 2008
Frank J Fabozzi, Harry M Markowitz, and Francis Gupta. Portfolio selection.Handbook of finance, 2:3–13, 2008
work page 2008
-
[3]
Eugene F Fama and Kenneth R French. The capital asset pricing model: Theory and evidence.Journal of economic perspectives, 18(3):25–46, 2004
work page 2004
-
[4]
Value at risk.Financial analysts journal, 56(2):47–67, 2000
Thomas J Linsmeier and Neil D Pearson. Value at risk.Financial analysts journal, 56(2):47–67, 2000
work page 2000
-
[5]
R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distributions.Journal of banking & finance, 26(7):1443–1471, 2002
work page 2002
-
[6]
Henrique Pinto Ramos, Marcelo Brutti Righi, Pablo Cristini Guedes, and Fernanda Maria Müller. A comparison of risk measures for portfolio optimization with cardinality constraints.Expert Systems with Applications, 228:120412, 2023
work page 2023
-
[7]
A simulation comparison of risk measures for portfolio optimization
Marcelo Brutti Righi and Denis Borenstein. A simulation comparison of risk measures for portfolio optimization. Finance Research Letters, 24:105–112, 2018
work page 2018
-
[8]
Zi Loke, Say Leng Goh, Graham Kendall, Salwani Abdullah, and Nasser Sabar. Portfolio optimisation problem: A taxonomic review of solution methodologies.IEEE Access, PP:1–1, 01 2023
work page 2023
-
[9]
Fifty years of portfolio optimization
Ahti Salo, Michalis Doumpos, Juuso Liesiö, and Constantin Zopounidis. Fifty years of portfolio optimization. European Journal of Operational Research, 318(1):1–18, 2024
work page 2024
-
[10]
Okkes Ertenlice and Can B Kalayci. A survey of swarm intelligence for portfolio optimization: Algorithms and applications.Swarm and evolutionary computation, 39:36–52, 2018
work page 2018
-
[11]
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio.Deep learning, volume 1. MIT press Cambridge, 2016
work page 2016
-
[12]
Thomas Fischer and Christopher Krauss. Deep learning with long short-term memory networks for financial market predictions.European journal of operational research, 270(2):654–669, 2018
work page 2018
-
[13]
Jyotirmayee Behera, Ajit Kumar Pasayat, Harekrushna Behera, and Pankaj Kumar. Prediction based mean-value- at-risk portfolio optimization using machine learning regression algorithms for multi-national stock markets. Engineering Applications of Artificial Intelligence, 120:105843, 2023
work page 2023
-
[14]
Deep reinforcement learning: A brief survey.IEEE signal processing magazine, 34(6):26–38, 2017
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey.IEEE signal processing magazine, 34(6):26–38, 2017. 20 APREPRINT- JULY9, 2026
work page 2017
-
[15]
Himanshu Choudhary, Arishi Orra, Kartik Sahoo, and Manoj Thakur. Risk-adjusted deep reinforcement learning for portfolio optimization: A multi-reward approach.International Journal of Computational Intelligence Systems, 18(1):126, 2025
work page 2025
-
[16]
Empirical asset pricing via machine learning.The Review of Financial Studies, 33(5):2223–2273, 2020
Shihao Gu, Bryan Kelly, and Dacheng Xiu. Empirical asset pricing via machine learning.The Review of Financial Studies, 33(5):2223–2273, 2020
work page 2020
-
[17]
Himanshu Choudhary, Arishi Orra, Manoj Thakur, Xiao-Zhi Gao, and Prabhat Kumar Sahu. A cvar-constrained safe reinforcement learning framework with action repair for practical portfolio optimization.IEEE Transactions on Artificial Intelligence, 2026
work page 2026
-
[18]
Portfolio selection.The Journal of Finance, 7(1):77–91, 1952
Harry Markowitz. Portfolio selection.The Journal of Finance, 7(1):77–91, 1952
work page 1952
-
[19]
Petter N Kolm, Reha Tütüncü, and Frank J Fabozzi. 60 years of portfolio optimization: Practical challenges and current trends.European Journal of Operational Research, 234(2):356–371, 2014
work page 2014
-
[20]
Suraj S Meghwani and Manoj Thakur. Multi-objective heuristic algorithms for practical portfolio optimization and rebalancing with transaction cost.Applied Soft Computing, 67:865–894, 2018
work page 2018
-
[21]
Huy Chau, Duy Nguyen, and Thai Nguyen. Continuous-time optimal investment with portfolio constraints: a reinforcement learning approach.European Journal of Operational Research, 2025
work page 2025
-
[22]
Capturing Financial markets to apply Deep Reinforcement Learning
Souradeep Chakraborty. Capturing financial markets to apply deep reinforcement learning.arXiv preprint arXiv:1907.04373, 2019
work page internal anchor Pith review Pith/arXiv arXiv 1907
-
[23]
Yuming Li, Pin Ni, and Victor Chang. Application of deep reinforcement learning in stock trading strategies and stock forecasting.Computing, 2019
work page 2019
-
[24]
Application of deep q-network in portfolio management
Ziming Gao, Yuan Gao, Yi Hu, Zhengyong Jiang, and Jionglong Su. Application of deep q-network in portfolio management. In2020 5th IEEE International Conference on Big Data Analytics (ICBDA), pages 268–275. IEEE, 2020
work page 2020
-
[25]
A framework of hierarchical deep q-network for portfolio management
Yuan Gao, Ziming Gao, Yi Hu, Sifan Song, Zhengyong Jiang, and Jionglong Su. A framework of hierarchical deep q-network for portfolio management. InICAART (2), pages 132–140, 2021
work page 2021
-
[26]
Deep reinforcement learning for portfolio selection.Global Finance Journal, 62:101016, 2024
Yifu Jiang, Jose Olmo, and Majed Atwi. Deep reinforcement learning for portfolio selection.Global Finance Journal, 62:101016, 2024
work page 2024
-
[27]
Shaghayegh Abolmakarem, Farshid Abdi, Kaveh Khalili-Damghani, and Hosein Didehkhani. Predictive multi- period multi-objective portfolio optimization based on higher order moments: Deep learning approach.Computers & industrial engineering, 183:109450, 2023
work page 2023
-
[28]
Philip Ndikum and Serge Ndikum. Advancing investment frontiers: Industry-grade deep reinforcement learning for portfolio optimization.arXiv preprint arXiv:2403.07916, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[29]
Reinforcement learning for deep portfolio optimization.Electronic Research Archive, 32(9):5176, 2024
Ruyu Yan, Jiafei Jin, and Kun Han. Reinforcement learning for deep portfolio optimization.Electronic Research Archive, 32(9):5176, 2024
work page 2024
-
[30]
Junkyu Jang and NohYoon Seong. Deep reinforcement learning for stock portfolio optimization by connecting with modern portfolio theory.Expert Systems with Applications, 218:119556, 2023
work page 2023
-
[31]
Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization
Pengqian Yu, Joon Sern Lee, Ilya Kulyatin, Zekun Shi, and Sakyasingha Dasgupta. Model-based deep reinforce- ment learning for dynamic portfolio optimization.arXiv preprint arXiv:1901.08740, 2019
work page internal anchor Pith review Pith/arXiv arXiv 1901
-
[32]
Raghu Nandan Sengupta, Aditya Gupta, Subhankar Mukherjee, and Gregor Weiss. Bi-objective reliability based optimization: an application to investment analysis.Annals of Operations Research, 333(1):47–78, 2024
work page 2024
-
[33]
Weifei Hu, Sichuang Cheng, Jiquan Yan, Jin Cheng, Xiang Peng, Hyunkyoo Cho, and Ikjin Lee. Reliability-based design optimization: a state-of-the-art review of its methodologies, applications, and challenges.Structural and Multidisciplinary Optimization, 67(9):168, 2024
work page 2024
-
[34]
Reliability in portfolio optimization using uncertain estimates.Sankhya B, 85(Suppl 1):199–233, 2023
Raghu Nandan Sengupta, Rachit Seth, and Peter Winker. Reliability in portfolio optimization using uncertain estimates.Sankhya B, 85(Suppl 1):199–233, 2023
work page 2023
-
[35]
P Jana, TK Roy, and SK Mazumder. Multi-objective possibilistic model for portfolio selection with transaction cost.Journal of computational and applied mathematics, 228(1):188–196, 2009
work page 2009
-
[36]
Wei Chen. Artificial bee colony algorithm for constrained possibilistic portfolio optimization problem.Physica A: Statistical Mechanics and its Applications, 429:125–139, 2015
work page 2015
-
[37]
Proximal Policy Optimization Algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.ArXiv, abs/1707.06347, 2017. 21 APREPRINT- JULY9, 2026 A Appendix Table A1: Portfolio Weights Comparison Across Market Regimes – Variance Optimization Pre-COVID COVID Post-COVID Asset NSGA-II PPO NSGA-II PPO NSGA-II PPO SSE 3.0 6.7 25.7 7...
work page internal anchor Pith review Pith/arXiv arXiv 2017
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.