Pith. sign in

REVIEW 4 major objections 4 minor 48 references

SafeOR-Gym: A Benchmark Suite for Safe Reinforcement Learning Algorithms on Practical Operations Research Problems

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SafeOR-Gym supplies nine operations-research benchmark environments, integrated with a constrained-MDP interface, and its evaluation shows that current safe-RL algorithms solve some of them but fail on the mixed-integer and nonconvex ones.

desk verdict Useful benchmark suite, but the Gurobi 'optimal reward' ground truth is not comparable to the RL reward, and there are sign errors in the tables. read the letter →

arxiv 2506.02255 v2 pith:XB6HVMJH submitted 2025-06-02 cs.LG

Asha Ramanujam (1) , Adam Elyoumi (1) , Hao Chen (1) , Sai Madhukiran Kompalli (1) , Akshdeep Singh Ahluwalia (1) , Shraman Pal (1) , Dimitri J. Papageorgiou (2) , Can Li (1) ((1) Davidson School of Chemical Engineering
show 7 more authors
Purdue University West Lafayette IN (2) Energy Sciences ExxonMobil Technology Engineering Company Annandale NJ)
This is my paper · ORCID
classification cs.LG
keywords safereinforcementlearningconstrainedMarkovdecisionprocessesoperationsresearchbenchmarksCMDPenvironmentsmixed-integerdecisionsnonconvexconstraintsplanningandschedulingenergysystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SafeOR-Gym argues that safe reinforcement learning needs benchmarks drawn from operations research rather than only robot control, and it provides nine such environments. Each environment encodes a realistic planning, scheduling, or power-system problem as a constrained Markov decision process, with separate cost signals for constraint violations, finite horizons, and hybrid discrete–continuous actions. The authors test five safe-RL algorithms against solver-computed optimal rewards and find a wide spread: some tasks, such as unit commitment and inventory management, are learnable by at least one method, while multiperiod blending, resource task networks, and grid-integrated storage defeat all tested methods. The paper's claim that matters is that current safe-RL methods, especially soft-penalty and off-policy Lagrangian approaches, hit fundamental limits on mixed-integer and nonconvex OR problems, and that a practical benchmark is needed to push past them.

What carries the argument

The load-bearing object is the constrained Markov decision process (CMDP): each environment exposes reward plus a separate cost signal that penalizes infeasible actions or state violations, and safety is measured by that cost. Around this interface, the suite adds two mechanisms. First, every environment defines a deterministic optimization model of the underlying OR problem, so an external mixed-integer/nonconvex solver can compute an 'optimal reward' baseline that gives the benchmark a ground-truth column. Second, transition dynamics include action sanitization, such as clipping, rounding, or forced repairs, so the agent sees a feasible state regardless of its action, with the deviation logged as cost; this separates 'cost of violating constraints' from 'reward earned.' The comparison that carries the argument is the gap between each algorithm's evaluation reward and cost and the solver's optimal reward.

What would settle it

Take the blending, resource-task-network, and grid-storage instances and re-run the five algorithms under three penalty-coefficient scalings while verifying the solver's reported optimum with a zero-gap tolerance. If any algorithm reaches the recomputed optimum under some weighting, or if the reported optimal reward moves materially when penalty weights change, the claim that these tasks expose fundamental limitations of current safe-RL algorithms would be put in doubt.

Watch

Extended reading notes

Core claim

SafeOR-Gym's central claim is that realistic OR problems, formulated as constrained Markov decision processes, expose a limitation that control-oriented safe-RL benchmarks miss: structured, cost-based constraint violations and mixed discrete–continuous actions. The paper builds nine environments, including resource and state task networks, unit commitment, generation and transmission expansion, multiperiod blending, multi-echelon inventory, grid-integrated storage, integrated scheduling and maintenance, and air-separation-unit scheduling, and scores five algorithms against optimal rewards computed by an optimization solver on the underlying deterministic models. Its headline finding is that projection-based trust-region methods and one constraint-rectified method usually reach low-cost near-optimal policies on several tasks, while an off-policy Lagrangian method consistently underperforms and every tested method fails to learn reasonable policies on the blending and task-network environments. The authors attribute the hard cases to nonconvexity and mixed-integer structure rather than to any single algorithm.

Load-bearing premise

The whole evaluation depends on the assumption that the solver's solution is the true best possible reward for each environment and that the authors' hand-chosen penalty coefficients correctly represent how much safety should cost; if either is wrong, the performance gaps the paper reports are not a fair measure of the algorithms.

Editorial extensions

If this is right

  • At least one tested algorithm learns a policy within roughly 10 to 35 percent of the solver-optimal reward, at low cost, on unit commitment, inventory management, expansion planning, and maintenance scheduling, so these environments can serve as positive controls for future safe-RL research.
  • No tested algorithm reaches reasonable-optimality on multiperiod blending, resource or state task networks, or grid-integrated storage, supporting the paper's conclusion that mixed-integer and nonconvex constraints are the hardest current barrier.
  • The off-policy Lagrangian baseline consistently has the highest costs and lowest rewards, while on-policy trust-region and constraint-rectified methods most often top the leaderboard, indicating that the update mechanism matters for cost compliance.
  • The penalty-based PPO method's large train-versus-evaluation gaps in several environments show that adaptive-penalty schedules can look good during training yet fail at deployment, so evaluation-time costs must be reported alongside rewards.
  • Solver-computed optimal rewards turn each environment into a quantified optimality-gap benchmark, letting future work measure progress as distance to the optimum instead of relative algorithm ranking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to sweep the penalty coefficients that define each environment's cost signal; if some weighting lets a previously failing algorithm reach near-optimal reward, part of the reported failure is reward shaping rather than algorithmic limitation.
  • Because the environments already sanitize actions by clipping, rounding, or forced feasibility repairs, a natural experiment is to compare the soft-penalty formulations against hard action-constrained policies, which would isolate whether the bottleneck is constraint representation or policy optimization.
  • The suite's deterministic instances with solver-computed optima could be given stochastic demand or renewable forecasts; if the current failure pattern persists, the difficulty is structural, whereas if it changes, the deterministic ground-truth setup was masking part of the problem.
  • Reporting the fraction of episodes in which each agent chooses the zero or do-nothing action would test whether high violation penalties are teaching inaction in the blending and task-network environments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. SafeOR-Gym introduces a suite of nine operations-research-inspired environments for safe reinforcement learning, implemented with a CMDP interface in OmniSafe. The paper provides detailed environment descriptions, evaluates five safe RL algorithms (CPO, DDPGLag, OnCRPO, P3O, TRPOLag) on these environments, and reports that some environments are tractable while others expose fundamental limitations of current methods. The central claim is that the suite serves as a challenging, practical benchmark for safe RL in structured, constraint-rich industrial decision problems.

Significance. If the evaluation methodology were sound, SafeOR-Gym would fill a real gap: existing safe RL benchmarks are dominated by robotic control, while OR problems offer structured constraints, mixed-integer decisions, and long horizons. The paper ships detailed environment descriptions, integration with OmniSafe, and reproducible code, which are valuable assets for the community. The empirical finding that projection-based on-policy methods (CPO, TRPOLag, OnCRPO) generally outperform DDPGLag across these tasks is plausible and interesting. However, the paper's headline quantitative claims about 'reasonable-optimality' and performance gaps depend critically on the comparability of the computed optimal rewards, and that comparability is not established.

major comments (4)
  1. [§4.1, 'Optimal reward of the environments'; §A.4.7] The claim that Gurobi solves the environment's optimization problem to global optimality and that the resulting reward is the 'ground truth' upper bound for safe RL policies is not supported. The RL environment's reward is a hand-defined economic objective that differs from the hard-constrained optimization model solved by Gurobi. For GTEPEnv, the reward in §A.4.7 includes only installation costs, while demand violations are relegated to the separate cost signal; the Gurobi model enforces hard constraints and does not include the same penalty coefficients (λ0, λ2). Consequently, the reported optimal reward is not an upper bound on the achievable RL reward. Table 1 confirms this impossibility: DDPGLag attains a GTEPEnv reward of -19 with a cost of 689140, far above the reported optimal of -267.7. The paper never explains this inconsistency, yet uses the optimum to classify environments and measure 'reasonable-optimality' gaps. This is load-bearing for the central evaluation claim and must be fixed, either by reformulating the optimization model to match the RL reward exactly, or by clearly defining the optimum as the reward of the optimal feasible policy under the same penalty structure.
  2. [Table 1 vs Table 2, BlendingEnv] There is a direct sign inconsistency in the reported optimal reward for BlendingEnv. Table 1 lists the optimal reward as +1800, while Table 2 lists the optimal reward as -1800 for the same underlying optimization problem (with different action-correction strategies). If this is intentional (e.g., a sign convention for minimization versus maximization), it is never stated, and it makes the reported gaps in the two tables impossible to compare. This inconsistency undermines the quantitative claims about performance gaps for BlendingEnv and its variants.
  3. [§4.1, 'Evaluation Criteria'] The definitions of 'best' and 'worst' performing algorithms, 'significant gap' (absolute difference > 100 and relative > 30%), and 'reasonable-optimality' (gap < 35%) are ad hoc. These thresholds are used to divide environments into 'tractable' versus 'not trained to reasonable optimality,' which is a central conclusion of the paper. No justification or sensitivity analysis is provided. For example, changing the 35% threshold or the 'within five times the lowest cost' filter could change which algorithms are labeled best or worst in several environments. Since these criteria are not derived from any principle, the qualitative taxonomy of environments rests on arbitrary cutoffs.
  4. [§3.4 vs §4.1; §A.6.5] The manuscript calls InvMgmtEnv a stochastic environment with demand drawn from a normal distribution (in §3.4 and §A.6.5), but §4.1 states that evaluation results show 'consistent performance in the determinstic environments' and uses the determinism of the underlying OR problems to justify computing a single optimal reward from Gurobi. This is internally inconsistent. Either InvMgmtEnv is stochastic and the determinism claim must be qualified, or the demand distribution in the environment is not actually sampled during evaluation. The paper also does not report the random seed or the number of evaluation samples per environment beyond '10 episodes,' so the statistical meaning of the reported rewards and costs for stochastic environments is unclear.
minor comments (4)
  1. [Throughout] There are numerous typos and misspellings that should be corrected: 'tranmision capacity' (§3.2), 'Upate electricity prices' (§3.6), 'determinstic' (§4.1), 'OminiSafe' (Abstract, §1, §4), 'envrionments' (§4.1), 'opeartions research' (§4.1), and 'evalute' (§4.1 in the same sentence).
  2. [§4.1, Figure 1] The figure caption lists a legend with 'Best,' but it is unclear what 'Best' denotes in the training curves, since the curves show individual algorithms. If 'Best' refers to the optimal reward, it should be labeled explicitly and shown as a constant line; otherwise, remove it from the legend.
  3. [§A.5.6] In the cost function for BlendingEnv, the demand-bound violation formula uses 'b lb_s' and 'I b_new,j' in places where the demand inventory variables and bounds should appear. These appear to be copy-paste errors from the blender formulas and make the formal definition of the demand violation cost ambiguous.
  4. [§B.3, Table 3] The table name says 'Wall-clock training time (hours),' but some entries are below one hour (e.g., RTNEnv: 0.12). If these are hours, they imply very short training runs; if they are not, the units should be clarified. The discussion in §B.3 refers to 'identical runtimes' and 'roughly double the wall-clock time,' but the table itself does not indicate the number of epochs or seeds per experiment.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark environments and optimal baselines are built from external OR models and an independent solver, not from the algorithm outputs.

full rationale

No circularity is found. SafeOR-Gym is a benchmark construction, not a derivation: each environment is built from external OR formulations (RTN/STN, unit commitment, blending, etc.) and the evaluation compares standard safe-RL algorithms against optimal values obtained by the independent Gurobi solver on deterministic optimization counterparts. The optimal rewards are therefore not computed from the algorithm outputs, and no fitted parameter is renamed as a prediction. The paper's self-citations (e.g., [16], [20], [23]) are background references for problem formulations and do not carry the benchmark's conclusions. Apparent inconsistencies, such as DDPGLag's GTEPEnv reward (-19) exceeding the reported optimum (-267.7) and the BlendingEnv optimum sign change between Table 1 and Table 2, point to a possible mismatch between the RL reward/cost model and the hard-constrained Gurobi model; that is a validity or correctness concern, not a circularity, because no equation in the paper reduces a claimed result to its own input.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central contribution is a set of environments and an evaluation, not a theoretical derivation. The main 'tuning knobs' are the hand-set penalty coefficients, action-correction strategies, and evaluation thresholds. There are no invented physical entities.

free parameters (4)
  • Penalty coefficients in GTEPEnv (lambda_0, lambda_2) = Hand-chosen, values not reported
    Control the trade-off between sparse installations and unmet-demand penalties; they shape the cost signal that the safe RL agent must satisfy.
  • Penalty coefficients in BlendingEnv (lambda_B, lambda_0,B, lambda_0,M, lambda_0,Q) = Hand-chosen, values not reported
    Set the cost of violating inventory bounds, in-out rules, and product property specs; these determine how much constraint violations are penalized relative to profit rewards.
  • Penalty factors in InvMgmtEnv and GridStorageEnv (phi_action, phi_on_hand, phi_pipeline, phi_bal, etc.) = Hand-chosen, values not reported
    Scale the per-unit violations of action bounds, inventory bounds, and power-balance residuals; the paper notes 'suboptimal tuning' of such parameters may explain poor performance in GridStorageEnv.
  • Evaluation selection thresholds (5x cost, 25 units, 10x, 100 units, 35% optimality gap) = 5, 25, 10, 100, 35%
    Ad hoc thresholds used to define 'best', 'worst', and 'significant gap' in the evaluation; they influence the paper's claims about which algorithms are tractable and which fail.
assumptions (4)
  • domain assumption Gurobi solves the environment's nonconvex optimization problems to global optimality while strictly enforcing all constraints (Section 4.1)
    This is the basis for the 'ground truth' optimal rewards used to judge algorithm performance. Gurobi's global solver can handle the small bilinear MIQCP instances, but the paper states it as a general capability without qualification.
  • domain assumption The cited OR models (RTN from Pantelides, UC from Knueven et al., blending from Chen and Maravelias, etc.) are faithful representations of real-world operations problems
    The realism of the benchmarks rests on these models; if the models are oversimplified, the suite's relevance to industrial practice is reduced.
  • domain assumption A CMDP cost signal, rather than a constraint that must be hard-satisfied, is an appropriate way to model safety for these problems
    The suite defines safety as penalized costs (soft constraints), but real industrial safety often involves hard feasibility. The paper acknowledges this in future work by calling for action-constrained RL.
  • domain assumption The specific instance sizes (e.g., 5 generators, 3 compressors, 30 days) are representative enough to draw conclusions about algorithm performance
    Small instances may not reflect the scaling challenges of real OR problems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SafeOR-Gym: A Benchmark Suite for Safe Reinforcement Learning Algorithms on Practical Operations Research Problems." pith.science (2026). https://pith.science/paper/XB6HVMJH

@misc{pith2026250602255,
  author       = {Pith},
  title        = {Pith review of: SafeOR-Gym: A Benchmark Suite for Safe Reinforcement Learning Algorithms on Practical Operations Research Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XB6HVMJH}},
  note         = {Machine review of arXiv:2506.02255}
}
read the original abstract

Most existing safe reinforcement learning (RL) benchmarks focus on robotics and control tasks, offering limited relevance to high-stakes domains that involve structured constraints, mixed-integer decisions, and industrial complexity. This gap hinders the advancement and deployment of safe RL in critical areas such as energy systems, manufacturing, and supply chains. To address this limitation, we present SafeOR-Gym, a benchmark suite of nine operations research (OR) environments tailored for safe RL under complex constraints. Each environment captures a realistic planning, scheduling, or control problems characterized by cost-based constraint violations, planning horizons, and hybrid discrete-continuous action spaces. The suite integrates seamlessly with the Constrained Markov Decision Process (CMDP) interface provided by OmniSafe. We evaluate several state-of-the-art safe RL algorithms across these environments, revealing a wide range of performance: while some tasks are tractable, others expose fundamental limitations in current approaches. SafeORGym provides a challenging and practical testbed that aims to catalyze future research in safe RL for real-world decision-making problems.

Figures

Figures reproduced from arXiv: 2506.02255 by the authors.

Figure 1
Figure 1. Training curves showing the average reward and cost per episode over training epochs [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Training curves of average reward and cost per episode across three additional environments [PITH_FULL_IMAGE:figures/full_fig_p047_2.png] view at source ↗
Figure 3
Figure 3. Average number of episode violations for different epochs for RTNEnv [PITH_FULL_IMAGE:figures/full_fig_p049_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Average number of episode violations for different epochs for STNEnv [PITH_FULL_IMAGE:figures/full_fig_p050_4.png]
Figure 5
Figure 5. Figure 5: Average number of episode violations for different epochs for UCEnv-v0 [PITH_FULL_IMAGE:figures/full_fig_p050_5.png]
Figure 6
Figure 6. Figure 6: Average number of episode violations for different epochs for UCEnv-v1 [PITH_FULL_IMAGE:figures/full_fig_p051_6.png]
Figure 7
Figure 7. Figure 7: Average number of episode violations for different epochs for GTEPEnv without transmis [PITH_FULL_IMAGE:figures/full_fig_p051_7.png]
Figure 8
Figure 8. Figure 8: Average number of episode violations for different epochs for BlendingEnv with prop [PITH_FULL_IMAGE:figures/full_fig_p052_8.png]
Figure 9
Figure 9. Figure 9: Average number of episode violations for different epochs with BlendingEnv with disable [PITH_FULL_IMAGE:figures/full_fig_p052_9.png]
Figure 10
Figure 10. Figure 10: Average number of episode violations for different epochs for BlendingEnv with none [PITH_FULL_IMAGE:figures/full_fig_p053_10.png]
Figure 11
Figure 11. Figure 11: Average number of episode violations for different epochs for InvMgmtEnv [PITH_FULL_IMAGE:figures/full_fig_p054_11.png]
Figure 12
Figure 12. Figure 12: Average number of episode violations for different epochs for GridStorageEnv [PITH_FULL_IMAGE:figures/full_fig_p054_12.png]
Figure 13
Figure 13. Figure 13: Average number of episode violations for different epochs for SchedMaintEnv [PITH_FULL_IMAGE:figures/full_fig_p055_13.png]
Figure 14
Figure 14. Figure 14: Average number of episode violations for different epochs for ASUEnv [PITH_FULL_IMAGE:figures/full_fig_p056_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 32 canonical work pages

  1. [1]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learning Research, pages 22–31, 2017

    work page 2017

  2. [2]

    Routledge, 2021

    Eitan Altman.Constrained Markov decision processes. Routledge, 2021

    2021

  3. [3]

    PC-Gym: Benchmark Environments for Process Control Problems.arXiv preprint arXiv:2410.22093,

    Maximilian Bloor, José Torraca, Ilya Orson Sandoval, Akhil Ahmed, Martha White, Mehmet Mercangöz, Calvin Tsay, Ehecatl Antonio Del Rio Chanona, and Max Mowbray. PC-Gym: Benchmark Environments for Process Control Problems.arXiv preprint arXiv:2410.22093,

    arXiv

  4. [4]

    OpenAI Gym.arXiv preprint arXiv:1606.01540, 2016

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym.arXiv preprint arXiv:1606.01540, 2016

    arXiv 2016

  5. [5]

    Constante Flores, and Can Li

    Hao Chen, Gonzalo E. Constante Flores, and Can Li. Physics-informed neural networks with hard linear equality constraints.Computers & Chemical Engineering, 189:108764, 2024. ISSN 0098-1354. doi: https://doi.org/10.1016/j.compchemeng.2024.108764. URL https: //www.sciencedirect.com/science/article/pii/S0098135424001820

    arXiv 2024

  6. [6]

    Maravelias

    Yifu Chen and Christos T. Maravelias. Preprocessing algorithm and tightening constraints for multiperiod blend scheduling: cost minimization.Journal of Global Optimization, 77 (3):603–625, Jul 2020. ISSN 1573-2916. doi: 10.1007/s10898-020-00882-3. URL https: //doi.org/10.1007/s10898-020-00882-3

    work page doi:10.1007/s10898-020-00882-3 2020

  7. [7]

    A comprehensive survey on safe reinforcement learning

    Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437–1480, 2015

    2015

  8. [8]

    A review of safe reinforcement learning: Methods, theory and applications.arXiv preprint arXiv:2205.10330, 2022

    Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications.arXiv preprint arXiv:2205.10330, 2022

    arXiv 2022

Show all 48 references
  1. [9]

    Gurobi Optimizer Reference Manual, 2025

    Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2025. URL https://www. gurobi.com

    2025

  2. [10]

    Zhouchun Huang and Qipeng Phil Zheng. A multistage stochastic programming approach for preventive maintenance scheduling of GENCOs with natural gas contract.European Journal of Operational Research, 287(3):1036–1051, 2020. doi: 10.1016/j.ejor.2020.03.036

    2020 doi

  3. [11]

    Hubbs, Hector D

    Christian D. Hubbs, Hector D. Perez, Owais Sarwar, Nikolaos V . Sahinidis, Ignacio E. Gross- mann, and John M. Wassick. OR-Gym: A Reinforcement Learning Library for Operations Research Problems.arXiv preprint arXiv:2008.06319, 2020. doi: 10.48550/arXiv.2008.06319

    2008 doi arXiv

  4. [12]

    Efficient action-constrained reinforce- ment learning via acceptance-rejection method and augmented mdps.arXiv preprint arXiv:2503.12932, 2025

    Wei Hung, Shao-Hua Sun, and Ping-Chun Hsieh. Efficient action-constrained reinforce- ment learning via acceptance-rejection method and augmented mdps.arXiv preprint arXiv:2503.12932, 2025. URLhttps://arxiv.org/abs/2503.12932

    2025 arXiv

  5. [13]

    Safety gymnasium: A unified safe reinforcement learning benchmark.Advances in Neural Information Processing Systems, 36:18964–18993, 2023

    Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark.Advances in Neural Information Processing Systems, 36:18964–18993, 2023

    2023

  6. [14]

    Omnisafe: An infrastructure for accelerating safe reinforcement learning research.Journal of Machine Learning Research, 25(285):1–6, 2024

    Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research.Journal of Machine Learning Research, 25(285):1–6, 2024

    2024

  7. [15]

    On mixed-integer programming formulations for the unit commitment problem.INFORMS Journal on Computing, 32(4): 857–876, 2020

    Bernard Knueven, James Ostrowski, and Jean-Paul Watson. On mixed-integer programming formulations for the unit commitment problem.INFORMS Journal on Computing, 32(4): 857–876, 2020. doi: 10.1287/ijoc.2019.0944. 10

    2020

  8. [16]

    Can Li, Antonio J Conejo, Peng Liu, Benjamin P Omell, John D Siirola, and Ignacio E Grossmann. Mixed-integer linear programming models and algorithms for generation and transmission expansion planning of power systems.European Journal of Operational Research, 297(3):1071–1082, 2022

    2022

  9. [17]

    Lillicrap, Jonathan J

    Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning

  10. [18]

    Unified frameworks for optimal process planning and scheduling

    Constantinos C Pantelides. Unified frameworks for optimal process planning and scheduling. InProceedings on the second conference on foundations of computer aided operations, pages 253–274, 1994

    1994

  11. [19]

    Reinforcement learning for process control: Review and benchmark problems.International Journal of Control, Automation and Systems, 23(1):1–40, 2025

    Joonsoo Park, Hyein Jung, Jong Woo Kim, and Jong Min Lee. Reinforcement learning for process control: Review and benchmark problems.International Journal of Control, Automation and Systems, 23(1):1–40, 2025

    2025

  12. [20]

    Algorithmic approaches to inventory management optimization.Processes, 9(1):102, 2021

    Hector D Perez, Christian D Hubbs, Can Li, and Ignacio E Grossmann. Algorithmic approaches to inventory management optimization.Processes, 9(1):102, 2021

    2021

  13. [21]

    Re- inforcement learning for efficient power systems planning: A review of operational and ex- pansion strategies.Energies, 17(9), 2024

    Gabriel Pesántez, Wilian Guamán, José Córdova, Miguel Torres, and Pablo Benalcazar. Re- inforcement learning for efficient power systems planning: A review of operational and ex- pansion strategies.Energies, 17(9), 2024. ISSN 1996-1073. doi: 10.3390/en17092167. URL https://www...

    2024 doi

  14. [22]

    Long duration battery sizing, siting, and operation under wildfire risk using progressive hedging

    Ryan Piansky, Georgia Stinchfield, Alyssa Kody, Daniel K Molzahn, and Jean-Paul Watson. Long duration battery sizing, siting, and operation under wildfire risk using progressive hedging. arXiv preprint arXiv:2404.12296, 2024

    2024 arXiv

  15. [23]

    A tutorial on multi-time scale optimization models and algorithms

    Asha Ramanujam and Can Li. A tutorial on multi-time scale optimization models and algorithms. arXiv preprint arXiv:2502.20568, 2025

    2025 arXiv

  16. [24]

    Prentice Hall Upper Saddle River, NJ, 1998

    Ronald L Rardin and Ronald L Rardin.Optimization in operations research, volume 166. Prentice Hall Upper Saddle River, NJ, 1998

    1998

  17. [25]

    Benchmarking Safe Exploration in Deep Reinforcement Learning

    Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019

    2019

  18. [26]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei, editors,Proceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Machine Learning Research...

    2015

  19. [27]

    MuJoCo: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. InProc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5026–5033, 2012. doi: 10.1109/IROS.2012.6386109

    2012

  20. [28]

    Xenos, Georgios M

    Dionysios P. Xenos, Georgios M. Kopanos, Matteo Cicciotti, and Nina F. Thornhill. Operational optimization of networks of compressors considering condition-based maintenance.Computers & Chemical Engineering, 84:117–131, 2016. doi: 10.1016/j.compchemeng.2015.08.008

    2016 doi

  21. [29]

    Crpo: A new approach for safe reinforcement learning with convergence guarantee

    Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learni...

    2021

  22. [30]

    Sustaingym: Reinforcement learning environments for sustainable energy systems.Advances in Neural Information Processing Systems, 36:59464–59476, 2023

    Christopher Yeh, Victor Li, Rajeev Datta, Julio Arroyo, Nicolas Christianson, Chi Zhang, Yize Chen, Mohammad Mehdi Hosseini, Azarang Golmohammadi, Yuanyuan Shi, et al. Sustaingym: Reinforcement learning environments for sustainable energy systems.Advances in Neural Information...

    2023

  23. [31]

    Penalized proximal policy optimization for safe reinforcement learning

    Linrui Zhang, Li Shen, Long Yang, Shixiang Chen, Xueqian Wang, Bo Yuan, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, ...

    2022

  24. [32]

    Grossmann, Clara F

    Qi Zhang, Ignacio E. Grossmann, Clara F. Heuberger, Arul Sundaramoorthy, and Jose M. Pinto. Air separation with cryogenic energy storage: Optimal scheduling considering electric energy and reserve markets.AIChE Journal, 61(5):1547–1558, 2015. doi: 10.1002/aic.14730. 12 Supplem...

    2015 doi

  25. [35]

    Sanitize Action: Prevent resource violations and enforce equipment availability by calcu- lating the maximum inventory available for a resource, maximum batch size that can be processed based on inventory levels, and clipping between the batch size bounds accordingly. bi,r,t =...

  26. [36]

    Xr,t+1 =X r,t +p t,r,0 pt =p t−1,r,1:τmax ⊕ X i max{νi,r,0} ·afinal i,t (1)

    Pending Outputs: Add outputs of a task to the pending output buffer and update inventory of resources that are being delivered in the next timestep. Xr,t+1 =X r,t +p t,r,0 pt =p t−1,r,1:τmax ⊕ X i max{νi,r,0} ·afinal i,t (1)

  27. [37]

    A part of the cost is calculated based on this

    Inventory Enforcement: Ensures inventory bounds are not violated. A part of the cost is calculated based on this. Refer to A.1.6. Xr,t+1 = min(X max r ,max(X min r , Xr,t+1)) 14 A.1.6 Cost Function The total cost at each timesteptis given by: Ct =C lb t +C ub t +C eq t +λ sani...

  28. [38]

    Sanitize Action: Prevent state violations and enforce unit availability by calculating the maximum available inventory for each input state, the maximum feasible batch size given current inventories, and clipping between the allowed batch size bounds accordingly. bi,s,t = max(...

  29. [39]

    Xs,t+1 =X s,t +p t,s,0 pt+1,s =p t,s,1:τmax ⊕ X e X i max(νi,s,0)·a final i,e,t (2)

    Pending Outputs: Add the output states of tasks to the pending output buffer and update inventories of materials delivered at the current timestep. Xs,t+1 =X s,t +p t,s,0 pt+1,s =p t,s,1:τmax ⊕ X e X i max(νi,s,0)·a final i,e,t (2)

  30. [40]

    Violations of these bounds contribute to the constraint cost

    Inventory Enforcement: Enforces inventory bounds to prevent overflow or underflow. Violations of these bounds contribute to the constraint cost. Xs,t+1 = min(X max s ,max(X min s , Xs,t+1)) A.2.6 Cost Function The total cost at each timesteptis given by: Ct =C lb t +C ub t +C ...

  31. [41]

    33 Sales and Backlog Update.Sales are Sr,m,t = min Dr,m,t +B r,m,t−1, Ir,t , then Ir,t ←I r,t −S r,m,t, B r,m,t =D r,m,t +B r,m,t−1 −S r,m,t

    Demand Realization.For each(r, m)∈ RM, Dr,m,t ∼ N(µr,m, σr,m). 33 Sales and Backlog Update.Sales are Sr,m,t = min Dr,m,t +B r,m,t−1, Ir,t , then Ir,t ←I r,t −S r,m,t, B r,m,t =D r,m,t +B r,m,t−1 −S r,m,t. Demand Forecast Shift. Dr,m,t+1:t+k = Dr,m,t+1, . . . , Dr,m,t+k ∀(r, m)...

  32. [42]

    Let the resulting action vector be at = pg,t g∈G ∥ cn,t n∈N ∥ pd n,t n∈N ∥ ℓn,t n∈N ∥ θn,t n∈N \{1}, withθ 1,t ≡0

    Action decoding, clipping, and penalty logging.Each normalized component is mapped back into its physical range: apre i,t = anorm,i,t + 1 2 (amax i −a min i ) +a min i , and clipped to remain within bounds[a min i , amax i ]. Let the resulting action vector be at = pg,t g∈G ∥ ...

  33. [43]

    Battery state-of-charge update.The battery SOC at each busnevolves as: En,t+1 =γE n,t +ηc n,t − 1 η pd n,t, n∈ N

  34. [44]

    Load-shedding enforcement.Any load shedding exceeding the global maximum is clipped: ℓn,t ←min(ℓ n,t, dmax global), penalising excess with factorϕ shed

  35. [45]

    Power-flow calculation.Compute power flows from voltage angles, enforcing zero flow on de-energised lines: fℓ,t = (Bij(θi,t −θ j,t), ℓ /∈ Dt 0, ℓ∈ D t , ℓ= (i, j)

  36. [46]

    Slack generation calculation.Slack generation sn,t is computed to enforce exact network balance: sn,t = max n 0, dn,t −ℓ n,t − X g:BusGeneratorLink[g]=n pg,t +c n,t −p d n,t + X (i,n)∈L f(i,n),t − X (n,j)∈L f(n,j),t o . 37

  37. [47]

    The nodal power-balance residual is: ∆n,t =P n,t − X j∈N Bnj(θn,t −θ j,t), and the network-balance penalty is: C bal t =ϕ bal X n∈N |∆n,t|

    Net nodal-injection and power-balance penalty.Compute net nodal injection at each busn: Pn,t = X g:BusGeneratorLink[g]=n pg,t +s n,t −d n,t +ℓ n,t −c n,t +p d n,t. The nodal power-balance residual is: ∆n,t =P n,t − X j∈N Bnj(θn,t −θ j,t), and the network-balance penalty is: C ...

  38. [48]

    Work” (active production) and “Off

    Demand-forecast and observation reconstruction.Update the forecast window at each bus n: Dn,t:t+k−1 = (dn,t, dn,t+1, . . . , dn,min{t+k−1,T} ), padded with zeros beyond horizon T . Form the next state st+1 from normalized SOC, voltage- angle differences, loading ratios, flows,...

  39. [2019]

    URLhttps://arxiv.org/abs/1509.02971

    arXiv

  40. [2024]

    doi: 10.48550/arXiv.2410.22093

    doi arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.