Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Simulation-Driven Reinforcement Learning in Queuing Network Routing Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that a simulation-trained Dyna-DDPG agent with separate next-state and reward predictors learns queueing-network routing policies that remain effective under single-server disruptions and scale to 100-node networks.

desk verdict A solid engineering artifact with an unsupported core claim: the split-predictor improvement is never ablated, and the scaling evidence measures runs, not training time. read the letter →

arxiv 2507.18795 v1 pith:IZABN7VH submitted 2025-07-24 cs.AI

Fatima Al-Ani , Molly Wang , Jevon Charles , Aaron Ong , Joshua Forday , Vinayak Modi This is my paper · ORCID
classification cs.AI
keywords reinforcementlearningqueueingnetworksroutingoptimizationDyna-DDPGdeepdeterministicpolicygradientsimulation-driventrainingmanufacturingsystemsserverdisruptionrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a simulation-driven reinforcement-learning agent can learn good routing policies for queueing networks that keep performing when a server breaks. Its central algorithmic claim is that splitting the Dyna-DDPG environment model into two separate networks—one predicting the next state, one predicting the reward—makes training more stable and sample-efficient than the original single-model design. The authors back this with experiments on an 11-node manufacturing-style network showing the learned policy shifts routing probability away from a blocked queue, tolerates noise in inter-arrival times, and needs only one or two training runs to reach consistent decisions for networks up to 100 nodes. If correct, the framework offers a reusable path to train routing policies in simulation before live deployment, where mistakes are costly. The authors also report the reward trajectory over training and recommend around 300 episodes for smaller systems.

What carries the argument

The load-bearing object is the enhanced Dyna-DDPG agent, whose actor and critic networks are augmented by two separate predictive models: a next-state predictor and a reward predictor. In Dyna-style planning, the agent uses these predictors to hallucinate additional experiences from state-action pairs stored in the replay buffer, increasing sample efficiency without extra real interactions. A masking layer converts the continuous action vector into routing probabilities per queue, using the network's edge list, so each node's outgoing queues receive a share of jobs. The reward is computed as $\text{reward} = -D/R$, the negative average end-to-end delay divided by the throughput ratio, which pushes the agent to reduce delays while maximising completed jobs. Blockage exploration trains the agent on states where one server's service rate is set to infinity, teaching it to reroute around failures.

What would settle it

A deployment or high-fidelity simulation test in which two servers fail at once, or service times become stochastic, would settle the robustness claim: if the learned policy's throughput degrades sharply or it fails to reroute, the central claim that it maintains robust performance under disruptions is contradicted.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that an enhanced Dyna-DDPG agent with separate next-state and reward predictor networks learns effective continuous-action routing policies for Jackson-style queueing networks. The paper shows the agent assigns higher routing probability to faster service paths, drops routing probability to a blocked server from about 0.78 to about 0.05 after training with single-server blockages, and maintains an increasing throughput trend when inter-arrival noise is introduced at evaluation. It also reports that, measured by the maximum standard deviation of final transition probabilities across trained agents, one training run is sufficient for a 10-node network and two runs suffice for 50- and 100-node networks. The authors frame this as evidence that the framework scales to larger network sizes without proportional increases in training effort.

Load-bearing premise

The load-bearing premise is that the queueing simulation—infinite buffers, FIFO service, deterministic service rates, rational job-following, and only single-server blockages—faithfully represents the real manufacturing or communication system where the learned policy will be deployed.

Editorial extensions

If this is right

  • Routing decisions can be learned directly as continuous probabilities per queue, avoiding the coarse discretisation of tabular or discrete-action approaches.
  • A policy trained with single-server blockage exploration transfers to evaluation-time blockages: in the reported 11-node network the routing probability to a blocked queue drops sharply, so servers can be taken offline without manual reconfiguration.
  • The improved stability from separate predictor networks means fewer training runs are needed: one run for 10 nodes and two for 50- and 100-node networks, under the paper's variance-based robustness criterion.
  • Moderate noise in inter-arrival times at evaluation reduces throughput only slightly, with the increasing-throughput trend preserved, indicating the learned policy is not brittle to out-of-distribution arrival patterns.
  • Training duration can be chosen by observing reward convergence: the paper recommends roughly 300 episodes for smaller systems and more episodes as network complexity grows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the robustness result transfers beyond the simulator, the same recipe—separate predictors plus blockage exploration—could be applied to other queueing-network domains such as logistics or data routing, where live training is also risky.
  • Editorial inference: a natural testable extension the paper leaves implicit is multi-server blockages; since only single-server outages were trained, the framework's behaviour under simultaneous failures is unknown and is the most direct stress test of the robustness claim.
  • Editorial inference: the separate next-state and reward predictors could plausibly improve other Dyna-style continuous-control algorithms in high-dimensional environments, because the original single shared model is reported to be unstable in this setting.
  • Editorial inference: the scaling result to 100 nodes was limited by compute, so the claim of modest scaling effort should be re-tested on larger and looped topologies before being used as a general rule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes a simulation-driven reinforcement learning framework for routing optimization in queueing networks, targeting manufacturing and communication applications. The authors implement a DDPG agent augmented with Dyna-style planning, in which the original combined next-state/reward predictor is split into two separate neural networks. They build a configurable queueing-network environment, introduce blockage exploration to train on single-server failures, and provide an evaluation suite that includes startup-behavior, convergence, noise, robustness, and scaling analyses. The stated central claims are that the split-predictor architecture significantly improves stability and sample efficiency, that the learned policies remain robust under disruptions, and that training scales effectively to networks up to 100 nodes. The paper also emphasizes the software-engineering and reproducibility aspects of the released package.

Significance. If the central claims were fully supported, the framework would be a practical contribution: it offers a configurable queueing-network simulator, a continuous-action RL agent with Dyna-style planning, and an evaluation toolkit, all packaged with unit/integration tests and version-controlled dependencies. The reproducibility-oriented features (Poetry, pytest, configuration files) are commendable and are a genuine strength. However, the paper's headline algorithmic contribution, the split next-state and reward predictors, is not validated against any baseline, and the robustness and scaling evidence is either partly built into training or measures a different quantity than claimed. For these reasons the current significance is limited; the framework may be useful as engineering infrastructure, but the scientific claims require substantially stronger empirical support.

major comments (4)
  1. [0.4.1 and 0.12] The central claim that splitting the original Dyna-DDPG predictor into separate next-state and reward networks 'significantly improves stability and sample efficiency' is not supported by any baseline comparison or ablation. Section 0.4.1 asserts that the original algorithm led to unstable training and poor convergence, but no results for the original Dyna-DDPG, plain DDPG, or a single-predictor control are shown. The learning curves and loss plots in Section 0.12 show only the proposed architecture. To substantiate the main contribution, the authors should add controlled experiments with identical hyperparameters, network seeds, and environments, reporting reward trajectories, convergence episodes, wall-clock time, and sample counts for both the split and original predictors.
  2. [0.13 and 0.2.3] The scaling claim is not supported by the evidence presented. Section 0.2.3 states that 'we demonstrate in section 0.6.2 that the algorithm's training time scales well with larger queueing networks,' but Section 0.13, the performance-scaling section, reports the number of training instances needed to meet a decision-variance threshold (one instance for 10 nodes, two for 50 and 100 nodes). This does not measure training-time growth or sample complexity, and it is not the quantity promised. The authors should either measure and report actual training time or sample counts as a function of network size, or explicitly reframe the scaling claim in terms of the number of runs required for decision robustness.
  3. [0.4.4] The description of how a server is made non-functional is technically inconsistent: the paper states that a server is 'rendered non-functional by setting its service rate to infinity.' In queueing semantics, an infinite service rate makes a server maximally fast, not blocked or non-functional. This is load-bearing because the robustness experiments in Sections 0.9 and 0.13 depend on the blockage mechanism. Please clarify the actual implementation: if the code blocks the server through a different mechanism (e.g., zero service rate or removal from the network), the text and figures should describe that mechanism accurately.
  4. [0.9 and 0.4.4] The robustness claim is partially circular. The agent is deliberately trained on single-server blockage scenarios (Section 0.4.4), and the evaluation in Section 0.9 measures performance under exactly this same class of single-server blockages. The drop in routing probability after blocking therefore largely demonstrates that training achieved its intended effect, not that the policy generalizes to new disruption types. The paper acknowledges that only single-server blockages were considered, but the abstract and conclusions nevertheless claim robust performance under disruptions. To support this claim, the authors should evaluate on held-out disruptions, such as multiple simultaneous blocked servers, blockages at untrained nodes, or blockages of different durations, and report whether the policy degrades gracefully in those settings.
minor comments (5)
  1. [0.2.3 and 0.6.2] There is a section-reference error: Section 0.2.3 refers to Section 0.6.2 for the training-time scaling demonstration, but Section 0.6.2 is 'Code Formatting'; the relevant material is in Section 0.13.
  2. [References] The citation for Dyna-DDPG appears mismatched: the reference list entry is Zhang, Boehmer and Whiteson (2020) 'Deep residual reinforcement learning,' which is not an obvious source for the Dyna-DDPG algorithm. Please verify and correct this citation.
  3. [0.8] The sample-size equation n = (z * sigma / E)^2 uses sigma without a definition in the equation; the text later refers to 'the maximum standard deviation' but the symbol should be defined explicitly in the equation or immediately before it.
  4. [0.3.2] The sentence 'queues with greater probability process jobs faster than others' is imprecise: routing probability is not a service rate, and the agent's action vector is not itself a property of the queue. Please rephrase to avoid confusing routing probability with processing speed.
  5. [0.9 and Figures 5-7] Figures 5, 6, and 7 should state explicitly which node's routing probability is shown and how the blockage is implemented; the captions currently only say 'Before blocking' and 'After blocking,' which is insufficient for reproducibility.

Circularity Check

1 steps flagged · score 2.0 of 10

Low circularity: the disruption-robustness demonstration is partly in training distribution, while the central split-predictor claim is unsupported by a baseline but not circular.

  1. fitted input called prediction [Section 0.9, blockage coverage evaluation (Figures 5-7)]
    "After the agent being trained on various blockage scenarios, this probability significantly drops to only around 0.05. As a result, training under outage cases can enable the agent to deal with blockages effectively."

    The disruption-robustness result is evaluated on the same single-blockage scenarios that were deliberately included in training. Section 0.4.4 renders a single server non-functional during training, and Section 0.9 uses weight parameters to give blockage cases equal coverage. The observed post-blocking drop in routing probability is therefore a check that the policy matched its training distribution, not an independent prediction about novel disruptions. The paper's own conclusion ('training under outage cases can enable the agent to deal with blockages effectively') makes the output a direct consequence of the training input.

full rationale

No fitted parameter is relabeled as a prediction, no load-bearing self-citation or imported uniqueness theorem appears, and no known result is renamed. The only partially circular element is the disruption-robustness claim: because the agent was explicitly trained on single-server blockage scenarios (Section 0.4.4), the observed routing-probability drop after blocking (Section 0.9) is an in-training-distribution check rather than an out-of-sample prediction; the paper is transparent about this dependence. The central claim that splitting Dyna-DDPG's combined predictor into separate next-state and reward networks improves stability and sample efficiency is not circular, although it is an evidence gap because no baseline against the original single-network Dyna-DDPG or plain DDPG is reported. Similarly, Section 0.13's scaling evidence counts required training instances rather than training time despite Section 0.2.3 promising a training-time demonstration; that too is a missing-support gap, not a circularity. Overall circularity is therefore low.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

No new physical or conceptual entities are postulated. The learning algorithm, state representation, and reward are explicit design choices; the main free parameters are unreported hyperparameters plus the hand-chosen w1/w2 weights. The key unstated burden is simulation fidelity, which the paper assumes without external validation.

free parameters (6)
  • learning_rate = not reported
    Key DDPG hyperparameter tuned via WandB or Ray; final value not given.
  • batch_size = not reported
    Tuned through population-based training; no final value reported.
  • planning_steps = not reported
    Number of Dyna planning steps per replay sample; directly controls compute and any sample-efficiency gain.
  • tau = not reported
    Soft-update coefficient for target networks; affects training stability.
  • epsilon = not reported
    Exploration noise used in planning; affects exploration behavior.
  • w1 and w2 = 0.5 each
    Weights balancing normal vs blockage training scenarios, chosen by hand after the authors say 0.5 'has proven to perform well' (Section 0.9).
assumptions (6)
  • domain assumption Jackson network and M/M/1 queueing abstractions with Poisson arrivals and exponential service times describe the target manufacturing flows.
    Used in Section 0.1 to justify the simulator; no validation against real factory data is given.
  • domain assumption Dyna-style planning improves DDPG sample efficiency.
    Relied on from cited literature (Section 0.2.3); not re-derived or benchmarked in this paper.
  • domain assumption The queueing-tool simulation is faithful enough that policies transfer to real systems.
    Assumed in Section 0.4.2; this is load-bearing for the deployment claims.
  • domain assumption Infinite buffer, FIFO discipline, rational job-following, and arrival rate below service rate.
    Stated assumptions in Section 0.4.2; simplify implementation but limit realism.
  • domain assumption Service rates are deterministic.
    Stated in Section 0.4.2, though Section 0.12 uses exponential processing times, an inconsistency.
  • ad hoc to paper Training on single-server blockages is sufficient to confer robustness.
    Introduced in Section 0.4.4; the paper itself notes in Section 0.9 that multi-server blockages are not covered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simulation-Driven Reinforcement Learning in Queuing Network Routing Optimization." pith.science (2026). https://pith.science/paper/IZABN7VH

@misc{pith2026250718795,
  author       = {Pith},
  title        = {Pith review of: Simulation-Driven Reinforcement Learning in Queuing Network Routing Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZABN7VH}},
  note         = {Machine review of arXiv:2507.18795}
}
read the original abstract

This study focuses on the development of a simulation-driven reinforcement learning (RL) framework for optimizing routing decisions in complex queueing network systems, with a particular emphasis on manufacturing and communication applications. Recognizing the limitations of traditional queueing methods, which often struggle with dynamic, uncertain environments, we propose a robust RL approach leveraging Deep Deterministic Policy Gradient (DDPG) combined with Dyna-style planning (Dyna-DDPG). The framework includes a flexible and configurable simulation environment capable of modeling diverse queueing scenarios, disruptions, and unpredictable conditions. Our enhanced Dyna-DDPG implementation incorporates separate predictive models for next-state transitions and rewards, significantly improving stability and sample efficiency. Comprehensive experiments and rigorous evaluations demonstrate the framework's capability to rapidly learn effective routing policies that maintain robust performance under disruptions and scale effectively to larger network sizes. Additionally, we highlight strong software engineering practices employed to ensure reproducibility and maintainability of the framework, enabling practical deployment in real-world scenarios.

Figures

Figures reproduced from arXiv: 2507.18795 by the authors.

Figure 1
Figure 1. Example edge list We can see that the queue type (or edge type) that is connecting node 0 and node 1 is Type 1. Additionally, the queue types that are connected to node 1 are Type 2, 3, and 4 which links to nodes 2, 3, 4 accordingly. Take node 1 for instance, to extract its routing probabilities from the action vector, we would extract the 2nd, 3rd, 4th elements in the vector. This conversion is achieved in get next… view at source ↗
Figure 2
Figure 2. Queueing network used for analysis 0.7 How do we remove or discount start-up behaviour in the simulation? In the given RL environment, the actions are initialized as equal transition probabilities to all possible next nodes. This initialisation causes the agent to be unstable at the beginning of training and it may take some time to converge to a stable state. Addressing the start-up behavior — also known as the ‘bu… view at source ↗
Figure 3
Figure 3. Reward start-up behaviour 0.8 Should we run many instances of the simulation to test a single decision? To ensure that the final policy is robust and has low uncertainty around its final decisions, it is essential to run multiple instances of training. However, this requires substantial computational resources. To determine the minimum number of instances needed to achieve a specified level of robustness, we have in… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Standard deviation against number of trained agents 0.9 How much learning coverage should we get of key states or peripheral states in our system? On one hand, key states are those that are frequently visited and critical for the success of the policy being learned. Si…
Figure 5
Figure 5. Figure 5: Before blocking [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 8
Figure 8. Figure 8: Evaluated reward against number of episodes trained for From [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Throughput rate in standard and noisy environments 12 [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Transition probabilities from node 1 The plot reward() function is used to demonstrate the agent’s reward per time step for a given episode — the last episode by default. This plot shows how the agent’s learning progresses during an episode. As the episode progresses,…
Figure 11
Figure 11. Figure 11: Reward per time step [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 13
Figure 13. Figure 13: , the actor initially experiences a high loss, which gradually stabilizes and converges to a lower policy loss, indicating that the agent is learning to maximize the Q values effectively [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 15
Figure 15. Figure 15: Reward Model loss [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Computation of Blackwell Optimal Policies using Rational Functions

    cs.AI 2025-08 conditional novelty 6.0 of 10

    Using symbolic comparisons of rational value functions near gamma=1, the authors obtain the first strongly polynomial algorithms for Blackwell-optimal policies in deterministic MDPs and a subexponential expected algor...

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    & Yao, D

    Buzacott, J. & Yao, D. (1986), ‘On queueing network models of flexible manufacturing systems’, Queueing Systems 1, 5–27. URL: https://link.springer.com/article/10.1007/BF01149326 Chen, C. & Tiong, R. (2018), ‘Using queuing theory and simulated annealing to design the facility layout in an agv-based modular manufacturing system’, International Journal of P...

    work page doi:10.1007/bf01149326 1986

  2. [2024]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S. & Hassabis, D. (2015), ‘Human-level control through deep reinforcement learning’, Nature 518, 529–533. URL: htt...

    arXiv 2015

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.