Learning to replenish: A hybrid deep reinforcement learning for dynamic inventory management in the pharmaceutical supply chains

Amandeep Kaur; Gyan Prakash

arxiv: 2606.06201 · v1 · pith:QDJLYZESnew · submitted 2026-06-04 · 💻 cs.AI

Learning to replenish: A hybrid deep reinforcement learning for dynamic inventory management in the pharmaceutical supply chains

Amandeep Kaur , Gyan Prakash This is my paper

Pith reviewed 2026-06-28 01:14 UTC · model grok-4.3

classification 💻 cs.AI

keywords deep reinforcement learninginventory managementpharmaceutical supply chainMarkov decision processreplenishment policystochastic demandshelf life constraints

0 comments

The pith

A hybrid A3C DPPO deep reinforcement learning algorithm learns adaptive replenishment policies that lower inventory costs in pharmaceutical supply chains facing uncertain demand and variable lead times.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper models pharmaceutical inventory management as a Markov decision process whose state tracks demand patterns, lead times, and shelf-life limits. It introduces a hybrid asynchronous advantage actor critic distributed proximal policy optimization algorithm to output order quantities in continuous space. Numerical experiments show the learned policy adapts to dynamic conditions and produces lower total costs than several benchmark methods. The same approach is tested on real pharmaceutical inventory records to check real-world applicability. A reader would care because improved replenishment rules could reduce both stockouts and expired-product waste in time-sensitive medical supply chains.

Core claim

The hybrid A3C DPPO algorithm, applied after formulating the replenishment task as an MDP, produces policies that adaptively adjust order quantities under stochastic demand and lead times, yielding lower overall inventory costs than standard benchmarks while preserving high service levels; validation on real pharmaceutical data confirms the policies remain feasible outside simulation.

What carries the argument

The hybrid A3C DPPO algorithm, a deep reinforcement learning method that merges asynchronous advantage actor-critic with distributed proximal policy optimization to output continuous order quantities.

If this is right

The learned policy automatically revises order quantities when demand patterns or lead times shift.
Total inventory holding, ordering, and shortage costs fall below those of fixed or heuristic replenishment rules.
Service levels stay high even as costs decline because the objective balances both goals.
The continuous-action formulation allows direct use of real order quantities instead of discretized approximations.
Numerical results on actual pharmaceutical records indicate the approach transfers from simulation to observed data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same MDP-plus-DRL structure could apply to other perishable-goods chains such as food or blood products.
Adding online updating of the policy from live sales data might further reduce lag in adapting to new demand regimes.
If lead-time distributions change faster than the MDP state can track, performance gains would shrink unless the state is expanded.
Scaling the method to multi-echelon networks with shared warehouses would require checking whether the single-agent formulation still suffices.

Load-bearing premise

All relevant uncertainty in demand, lead times, and product shelf life can be captured inside the Markov decision process state so that the learned policy faces no unmodeled dynamics.

What would settle it

Running the trained A3C DPPO policy on the same real-world pharmaceutical dataset and obtaining higher total costs than a simple (s, S) policy or other listed benchmarks would falsify the claim of cost superiority.

read the original abstract

Pharmaceutical supply chains (PSCs) struggle with inventory management (IM) due to unpredictable demand patterns and variable lead times associated with restocking. This complexity is further compounded by the finite shelf lives of pharmaceutical products, which necessitate a delicate balance between adequate stock and minimal waste. These intertwined factors create a complex optimization problem that requires sophisticated inventory strategies to ensure both product availability and PSC efficiency. This study aims to develop an optimal inventory replenishment policy for pharmaceutical products that can handle the stochasticity arising from uncertain demand and variable PSC conditions. The objective is to maximize the profitability of the PSC while maintaining a high patient service level. We formulate the problem as a Markov decision process and propose a deep reinforcement learning (DRL) approach, specifically, a hybrid asynchronous advantage actor critic distributed proximal policy optimization (A3C DPPO)algorithm. The A3C DPPO algorithm is tailored to handle the continuous action space inherent in IM. The numerical results demonstrate that the proposed algorithm adaptively updates the inventory replenishment strategy under dynamic scenarios, resulting in lower inventory costs compared to various benchmarks. We also conduct numerical validation using real-world pharmaceutical inventory data to confirm the practical feasibility of the proposed algorithm.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper applies a hybrid A3C-DPPO method to pharmaceutical inventory replenishment and claims cost reductions on real data, but the evidence remains thin without numbers or implementation details.

read the letter

The main takeaway is that this work takes an existing hybrid actor-critic reinforcement learning setup and applies it to inventory control in pharmaceutical supply chains, where shelf life and service levels add constraints. It models the problem as an MDP and reports that the policy adapts to dynamic demand and lead times better than benchmarks, with some validation on actual industry data.

What stands out as new is the tailoring of A3C DPPO to continuous action spaces for replenishment quantities while explicitly handling perishability and patient service requirements. Earlier DRL inventory papers exist, but the pharma focus with its regulatory stakes is a reasonable domain extension. The paper does well by including real-world data validation rather than stopping at simulation, which helps show the approach can move beyond theory.

The soft spots sit in the results section. The abstract asserts lower costs without giving magnitudes, baseline definitions, statistical tests, or hyperparameter settings, so it is hard to judge whether the gains are meaningful or sensitive to tuning. The MDP assumption that the state captures all relevant stochasticity also looks optimistic if demand has unmodeled seasonal or external drivers that could break policy performance in practice.

This paper is for readers working on RL applications in operations research or supply chain analytics, particularly those interested in perishable goods. It offers a concrete case study rather than a new framework.

I would send it for peer review. The domain is relevant, the real-data step is a strength worth checking, and the method is established enough that referees can focus on whether the experiments hold up.

Referee Report

3 major / 0 minor

Summary. The manuscript formulates pharmaceutical inventory replenishment as a Markov decision process and proposes a hybrid A3C-DPPO deep reinforcement learning algorithm to learn adaptive policies that minimize costs while maintaining service levels under stochastic demand, variable lead times, and finite shelf lives. It claims that the algorithm produces lower inventory costs than benchmarks in dynamic simulated scenarios and demonstrates practical feasibility via validation on real-world pharmaceutical data.

Significance. If the claimed cost reductions and real-data validation are substantiated with full experimental details, the work could contribute a practical DRL method for perishable inventory optimization in regulated supply chains. The choice of a hybrid actor-critic approach for continuous replenishment quantities is a reasonable technical fit for the action space.

major comments (3)

[Abstract] Abstract: the central claim that the algorithm 'resulting in lower inventory costs compared to various benchmarks' is presented without any quantitative results (e.g., cost values, percentage reductions, benchmark definitions, or statistical significance), making it impossible to evaluate whether the performance gains are supported by the data or method.
The MDP formulation is asserted to capture all relevant stochasticity from demand, lead times, and shelf-life constraints, yet no definition of the state space, action space, transition dynamics, or reward function is supplied. This is load-bearing for the claim that the DRL agent learns an optimal policy without unmodeled dynamics.
No hyperparameter settings, training procedure, number of episodes, simulation details, or benchmark policies are described, preventing assessment of reproducibility or whether reported gains reduce to quantities fitted on the same data used for evaluation.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight areas where additional detail will strengthen the manuscript. We address each major comment below and will revise accordingly to improve transparency and reproducibility.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim that the algorithm 'resulting in lower inventory costs compared to various benchmarks' is presented without any quantitative results (e.g., cost values, percentage reductions, benchmark definitions, or statistical significance), making it impossible to evaluate whether the performance gains are supported by the data or method.

Authors: We agree that the abstract should provide quantitative support for the performance claims. In the revised manuscript, we will update the abstract to include specific metrics from the experimental results, such as percentage cost reductions relative to benchmarks, along with brief definitions of the benchmark policies and mention of statistical significance where applicable. revision: yes
Referee: The MDP formulation is asserted to capture all relevant stochasticity from demand, lead times, and shelf-life constraints, yet no definition of the state space, action space, transition dynamics, or reward function is supplied. This is load-bearing for the claim that the DRL agent learns an optimal policy without unmodeled dynamics.

Authors: We acknowledge the need for explicit definitions. The revised manuscript will expand the methodology section to fully specify the state space (inventory levels, pending orders, demand history, remaining shelf life), action space (continuous replenishment quantities), transition dynamics (stochastic demand, lead times, expiration), and reward function (inventory holding, shortage, and wastage costs with service level constraints). revision: yes
Referee: No hyperparameter settings, training procedure, number of episodes, simulation details, or benchmark policies are described, preventing assessment of reproducibility or whether reported gains reduce to quantities fitted on the same data used for evaluation.

Authors: We agree these details are required for reproducibility. The revised version will include a new subsection on experimental setup covering hyperparameter values, training procedure (including episode counts and convergence criteria), simulation environment parameters, and explicit descriptions of all benchmark policies (e.g., base-stock, (s,S) policies) along with how they were implemented and evaluated. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper formulates inventory replenishment as an MDP and applies a standard hybrid DRL algorithm (A3C-DPPO) to optimize policies under stochastic demand, lead times, and shelf-life constraints. Reported performance gains are obtained by running the learned policy on simulated dynamic scenarios and separate real-world pharmaceutical data, then comparing total costs against external benchmarks. No derivation step equates a claimed prediction to a fitted parameter by construction, no self-citation chain bears the central claim, and the MDP formulation is presented as an explicit modeling choice rather than a tautology. The numerical validation therefore remains an independent empirical test rather than a renaming or self-referential fit.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only abstract available, so ledger is necessarily incomplete. The approach rests on the standard assumption that inventory dynamics admit an MDP formulation and that DRL can optimize the resulting continuous-action policy without additional domain-specific constraints being violated.

pith-pipeline@v0.9.1-grok · 5741 in / 1172 out tokens · 20407 ms · 2026-06-28T01:14:35.436276+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

5 extracted references · 2 canonical work pages · 2 internal anchors

[1]

Ahmadi, E., Mosadegh, H., Maihami, R., Ghalehkhondabi, I., Sun, M., & Süer, G. A. (2022). Intelligent inventory management ap proaches for perishable pharmaceutical products in a healthcare supply chain. Computers & Operations Research, 147, 105968. Baloch, G., & Gzara, F. (2020). Capacity and assortment planning under one-way supplier-driven substitution...

2022
[2]

European Journal of Operational Research, 298(2), pp.401-412

Deep reinforcement learning for inventory control: A roadmap. European Journal of Operational Research, 298(2), pp.401-412. Bozkir, C. D. C., Kundakcioglu, O. E., & Henry, A. C. (2022). Hospital service levels during drug shortages: Stocking and tra nsshipment policies for pharmaceutical inventory. Journal of Global Optimization, 1-20. Candan, G., & Yazga...

2022
[3]

Kara, A., & Dogan, I. (2018). Reinforcement learning approaches for specifying ordering policies of perishable inventory syst ems. Expert Systems with Applications, 91, 150-158. Karamshetty, V., De Vries, H., Van Wassenhove, L. N., Dewilde, S., Minnaard, W., Ongarora, D., ... & Yadav, P. (2022). Inventory management practices in private healthcare facilit...

work page internal anchor Pith review Pith/arXiv arXiv 2018
[4]

Laínez, J

Socio-economic planning sciences, 87, 101602. Laínez, J. M., Schaefer, E., & Reklaitis, G. V. (2012). Challenges and opportunities in enterprise-wide optimization in the pharmaceutical industry. Computers & chemical engineering, 47, 19-28. Li, J., Liu, L., Hu, H., Zhao, Q., & Guo, L. (2018). An inventory model for deteriorating drugs with stochastic lead ...

2012
[5]

Lotfi, R., Rajabzadeh, M., Zamani, A., & Rajabi, M. S. (2022). Viable supply chain with vendor -managed inventory approach by considering blockchain, risk and robustness. Annals of Operations Research, 1-20. Masoudi, S., & Mirzazadeh, A. (2022). The new evidence reasoning based pharmaceutical inventory models with stochastic deteri oration rates and lead ...

work page internal anchor Pith review Pith/arXiv arXiv 2022

[1] [1]

Ahmadi, E., Mosadegh, H., Maihami, R., Ghalehkhondabi, I., Sun, M., & Süer, G. A. (2022). Intelligent inventory management ap proaches for perishable pharmaceutical products in a healthcare supply chain. Computers & Operations Research, 147, 105968. Baloch, G., & Gzara, F. (2020). Capacity and assortment planning under one-way supplier-driven substitution...

2022

[2] [2]

European Journal of Operational Research, 298(2), pp.401-412

Deep reinforcement learning for inventory control: A roadmap. European Journal of Operational Research, 298(2), pp.401-412. Bozkir, C. D. C., Kundakcioglu, O. E., & Henry, A. C. (2022). Hospital service levels during drug shortages: Stocking and tra nsshipment policies for pharmaceutical inventory. Journal of Global Optimization, 1-20. Candan, G., & Yazga...

2022

[3] [3]

Kara, A., & Dogan, I. (2018). Reinforcement learning approaches for specifying ordering policies of perishable inventory syst ems. Expert Systems with Applications, 91, 150-158. Karamshetty, V., De Vries, H., Van Wassenhove, L. N., Dewilde, S., Minnaard, W., Ongarora, D., ... & Yadav, P. (2022). Inventory management practices in private healthcare facilit...

work page internal anchor Pith review Pith/arXiv arXiv 2018

[4] [4]

Laínez, J

Socio-economic planning sciences, 87, 101602. Laínez, J. M., Schaefer, E., & Reklaitis, G. V. (2012). Challenges and opportunities in enterprise-wide optimization in the pharmaceutical industry. Computers & chemical engineering, 47, 19-28. Li, J., Liu, L., Hu, H., Zhao, Q., & Guo, L. (2018). An inventory model for deteriorating drugs with stochastic lead ...

2012

[5] [5]

Lotfi, R., Rajabzadeh, M., Zamani, A., & Rajabi, M. S. (2022). Viable supply chain with vendor -managed inventory approach by considering blockchain, risk and robustness. Annals of Operations Research, 1-20. Masoudi, S., & Mirzazadeh, A. (2022). The new evidence reasoning based pharmaceutical inventory models with stochastic deteri oration rates and lead ...

work page internal anchor Pith review Pith/arXiv arXiv 2022