{"id":"13a6aefb-0989-431b-8d99-eb3d033b7687","arxiv_id":"2411.18305","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Soft Actor-Critic agent trained with random delays outperformed replayed historical PID actions in an LSTM wastewater simulator on phosphorus removal, reward, and target deviation.","lead":"A deep reinforcement learning controller trained to handle random delays is reported to cut phosphorus pollution in a wastewater treatment simulator by 36% while reducing simulated costs. The result is a simulation-based benchmark, not a field test, and its strength depends on whether the simulator and historical control baseline are trustworthy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SAC-vs-PID comparison in Section 4.2 uses a replayed historical PID trace rather than a closed-loop PID on the simulator, so the reported 36%/55%/77%/9% improvements may reflect baseline mismatch rather than learned policy quality.","rationale":"The paper is transparent about the simulation setting and honestly states that results are in a simulated environment. The use of LSTM-based simulators and three delay scenarios is a reasonable design. However, the headline quantitative claims rest on a head-to-head comparison in Section 4.2 between SAC agents and a PID controller. The way that comparison is constructed fails to give the PID controller the same closed-loop feedback that SAC receives. Since the PID actions are historical traces, they are not reactions to the simulated state trajectory; any model error in the LSTM causes the simulated and historical states to diverge, and the replayed actions become increasingly inappropriate for the simulated system. This is a structural asymmetry, not a statistical or tuning issue. Even a perfect simulator would not fix it, because the replay baseline is open-loop by construction. The reader's concern about OOD simulator validity is real, but it is secondary: even if the LSTM is accurate for all actions, the baseline comparison is still unfair. The concrete test proposed above (closed-loop PID on the same simulator) directly addresses whether the reported improvements survive a fair baseline. If they do not, the central claim should be weakened to 'SAC can outperform a particular historical operating trace under this simulator,' which is a much narrower claim. We therefore recommend keeping the conditional verdict; the paper needs a closed-loop baseline comparison and simulator validation before the claim can be accepted.","tokens_in":13562,"tokens_out":4576,"duration_ms":41261,"concrete_test":"Implement a closed-loop PID controller on the LSTM simulator using the same phosphate setpoint, action bounds, and reward function, and evaluate it on the same September 15, 2022 test window. Compare its reward, target deviation, and total cost against SAC-RD in Table 2. If the closed-loop PID achieves materially better performance than the replayed trace (e.g., target deviation far below 24.17%), the reported SAC advantage is partly an artifact of the open-loop baseline. As a second check, run the historical PID actions through the simulator and compare the simulated state trajectory to the recorded historical trajectory; if they diverge, the replayed actions are not conditioned on the simulated states, confirming the mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.6, the authors state that the real PID controller was not accessible, so they 'assumed that the PID controller would make the same decisions as it did historically for the corresponding state' and used the historical action array aPID,t as the PID output. Both PID and SAC actions are then evaluated by sending them to the LSTM simulator to produce next states. This replay creates a systematic mismatch: the historical PID actions were responses to the actual plant's measured states, not to the simulated states generated by the LSTM. Once the LSTM trajectory diverges from the historical trajectory (even slightly), the replayed action at time t+1 is conditioned on a different state than the simulator's state at t+1. The PID is therefore not closing the loop on the simulated plant, while SAC actions are generated online from the simulated state. This biases the comparison against PID, independent of the OOD validity of the LSTM simulator. Table 2 and Figure 5, which support the headline reductions, are thus not a comparison between SAC and PID control but between SAC and a historical open-loop action trace. A correct closed-loop PID baseline, or a demonstration that the LSTM exactly reproduces the historical state trajectory under historical actions, is required before the central claim can be assessed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies the Soft Actor-Critic (SAC) algorithm to phosphorus removal control in a wastewater treatment plant (WWTP), using a custom LSTM-based simulator to model plant dynamics. The authors train SAC agents under three delay scenarios (no delay, constant delay, random delay) and compare the best agents against the existing plant PID controller. The headline results—reported in the abstract and Section 5.2—are that the random-delay agent achieves 36% lower phosphorus emissions, 55% higher reward, 77% lower target deviation, and 9% lower total costs relative to PID in the simulated environment. The comparison is performed by replaying historical PID actions through the LSTM simulator, as described in Section 3.6.","tokens_in":13848,"tokens_out":3204,"duration_ms":29612,"significance":"If the results are sound, the paper would provide a useful demonstration that delay-aware deep reinforcement learning can improve regulatory compliance and cost efficiency in a realistic WWTP control task. The manuscript is clearly written, the problem is well motivated, and the use of a simulator built from plant data is a sensible first step. However, the evaluation methodology as presented has load-bearing weaknesses: the PID baseline is not a closed-loop controller in the simulator, the LSTM simulator's generalization to SAC action sequences is not validated, and the results rest on a single evaluation day and single run. These issues prevent the current version from supporting the quantitative claims made in the abstract and conclusions.","major_comments":[{"comment":"The PID baseline is an open-loop replay of historical actions, not a closed-loop controller interacting with the simulated plant. The paper states in Section 3.6 that the real PID controller was not accessible and that the PID actions were taken directly from the dataset. Once the LSTM simulator's trajectory diverges from the historical trajectory—which it must for any nontrivial action sequence—the replayed PID action at time t+1 is conditioned on a state different from the simulator's state at t+1, while SAC actions are generated online from the simulated state. This biases the comparison in favor of SAC. The authors need to provide either a closed-loop PID baseline (e.g., a tuned PID implemented in the simulator) or demonstrate that the LSTM exactly reproduces the historical state trajectory under historical actions, so that the replay is faithful. Without this, the 36%/55%/77%/9% improvements cannot be attributed to policy quality.","section":"Section 3.6, Table 2, Figure 5"},{"comment":"The LSTM simulator's accuracy for the action sequences generated by the trained SAC policies is not validated. The simulator was trained on historical plant data, and SAC policies may produce dosing actions outside the historical distribution. The paper refers to prior work [11,12,13] for simulator development, but provides no error analysis, no multi-step prediction accuracy measures, and no sensitivity analysis with respect to action ranges in this manuscript. Since every head-to-head result in Table 2 and Figure 5 is computed inside this simulator, the authors should validate the simulator's multi-step predictions on action sequences similar to those produced by SAC (e.g., by comparing predicted versus actual plant responses on held-out data, or by reporting prediction errors for out-of-distribution actions). Without this, the reported gains may be artifacts of simulator inaccuracy.","section":"Section 3.2, Section 4.2"},{"comment":"The evaluation appears to rely on a single day (September 15, 2022) and a single run without error bars or multiple seeds. Figure 5 shows one trajectory and Table 2 reports one set of totals. Given the stochasticity in SAC training and in the random-delay environment, the reported improvements could be the result of a favorable seed or a particular day. The authors should report results across multiple evaluation days and multiple training seeds, with means and standard deviations, before drawing conclusions about the relative performance of SAC-RD and PID.","section":"Section 4.2, Figure 5, Table 2"},{"comment":"The nonlinear penalty function Pcoef(x) = a·e^{z·x+c} + d contains shape parameters a, z, c, and d, but their numerical values are never reported, and no sensitivity analysis is provided. Since the reward function in Eq. (13) is the criterion on which all agents are compared, the ranking of SAC-RD, SAC-CD, SAC-ND, and PID may depend on these parameters. The authors should report the parameter values and test the robustness of the comparison to reasonable variations of these parameters.","section":"Section 3.3.3, Eq. (13)"}],"minor_comments":[{"comment":"The sampling statement 'κt ∼ DiscreteUniform(αmin, κmax)' appears to contain a typo; 'αmin' should likely be 'κmin'.","section":"Algorithm 1, line 17"},{"comment":"The notation for observation delay is inconsistent: Eq. (1) uses d_s, while Section 3.5.1 uses ω. Please unify the notation.","section":"Section 2.1 and Section 3.5.1"},{"comment":"The abstract reports a '36% reduction in phosphorus emissions,' but Table 2 does not contain an explicit 'emissions' metric; this appears to refer to 'Avg. Target' (average phosphorus concentration). Please define the metric explicitly and ensure the abstract matches the table.","section":"Abstract and Table 2"},{"comment":"References [21] and [24] are duplicate entries for the same PyTorch paper; consolidate them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central comparison is self-referential in two ways: the simulator is built by the same authors in prior work, and the PID baseline is a historical replay from the same dataset used to train the simulator. The manuscript acknowledges the PID limitation but does not address the implications for the validity of the comparison. I recommend that the editor require the authors to either provide a closed-loop baseline or clearly restrict claims to 'the replayed historical PID trace' rather than 'traditional control methods.' The paper's novelty is also somewhat limited, as the delay-handling mechanism is an application of existing wrapper ideas rather than a new algorithmic contribution, but the application context is of interest to the journal's readership."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one if you care about RL for process control, but keep your guard up around the PID comparison. The genuinely useful finding is internal: under the same LSTM simulator, SAC trained with random delays (RD) beats constant-delay (CD) and no-delay (ND) versions. That ordering makes sense, the training curves in Figure 3 show a consistent gap after ~10M steps, and the reward function is reasonable. The delay-wrapper design, borrowed from Bouteiller et al., is cleanly integrated into a Gym environment.\n\nThe weak spot is exactly what the stress-test note flagged, and it does not go away on reading the paper: Section 3.6 says the real PID was not accessible, so they replayed the historical PID action array against the LSTM simulator. That is not a closed-loop baseline. The PID actions at each step were responses to the real plant's states, not to the simulator's states. Once the LSTM trajectory deviates from history, the replayed PID is acting on stale information. SAC, by contrast, gets the simulator's current state every step. So Table 2's \"36% lower emissions, 55% higher reward, 77% lower deviation, 9% lower cost\" is partly an artifact of comparing a closed-loop learned policy to an open-loop historical trace, not a fair SAC-vs-PID test.\n\nTwo more soft spots, in decreasing severity. First, the evaluation is a single day (September 15, 2022) with no error bars, no multiple seeds, no different periods. That is thin for a paper making percentage claims. Second, the LSTM simulator is the same authors' prior work, and there is no validation that it stays accurate for SAC's out-of-distribution dosing actions. The paper is honest that results are simulated, but the reader has to take the simulator's fidelity largely on faith. To their credit, they do admit in Section 5.2 that PID beats SAC-CD and SAC-ND on total cost, which suggests they are not hiding the trade-off.\n\nWho is this for? People working on RL for WWTP or delayed-feedback industrial control. The internal delay ablation is a worthwhile data point. The PID comparison is not publishable as-is. The fix is straightforward: run a properly tuned closed-loop PID on the same simulator, report multiple test windows and seeds, and validate or at least acknowledge the OOD limits of the LSTM. If those revisions are made, this could be a solid application paper.\n\nRecommendation: send it to peer review, but with a major-revision expectation. The core idea is legitimate and the delay ablation is informative; the load-bearing flaw is in the baseline, not in the concept.","headline":"The delay-aware SAC result is the real contribution, but the headline gains over PID are inflated by an open-loop historical baseline and a one-day simulation, so the paper needs a proper closed-loop comparator before the central claim is credible.","tokens_in":14385,"tokens_out":1308,"would_cite":false,"duration_ms":14939,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A delay-aware Soft Actor-Critic agent cuts phosphorus emissions by 36 percent and total costs by 9 percent versus PID control in a simulated wastewater treatment plant.","keywords":["deep reinforcement learning","Soft Actor-Critic","time delays","LSTM simulator","wastewater treatment","phosphorus removal","process control","PID comparison"],"falsifier":"Take the plant's logged dataset and, for each logged state-action pair, replay the action through the LSTM and compare its multi-step prediction against the state actually recorded at the corresponding future time, focusing on time steps where the dosing action lies in the upper tail of the historical distribution; if the prediction error grows with action novelty, the reported 36 percent emission reduction is not a stable result. A stronger test is a closed-loop trial in which the trained random-delay SAC policy sets the salt doses at the real plant for several weeks and effluent phosphate is compared with the PID baseline over the same inflow conditions.","tokens_in":13406,"feed_emoji":"💧","tokens_out":6638,"duration_ms":59507,"temperature":0.7,"pith_summary":"This paper claims that a Soft Actor-Critic reinforcement learning agent, trained on an LSTM-based simulator that reproduces the delayed feedback of a wastewater treatment plant, can control chemical phosphorus removal better than the plant's existing PID controller. Training the agent under random action and observation delays produced the strongest policy: in the simulated environment it reduced phosphorus emissions by 36 percent, lowered deviation from the regulatory target by 77 percent, improved cumulative reward by 55 percent, and cut total operational costs by 9 percent relative to the historical PID trace. The authors care because wastewater plants are slow, nonlinear, and subject to stochastic delays, so reactive PID controllers tend to overshoot or undershoot the phosphate target, and a controller that anticipates delays could reduce environmental risk while saving on chemical and tax costs.","feed_headline":"Random-delay AI cuts phosphorus emissions 36 percent in simulation","feed_subtitle":"Soft Actor-Critic agent also beats the plant's PID on regulatory compliance and total cost in an LSTM-based wastewater testbed.","key_machinery":"The machinery has three parts. The learning core is Soft Actor-Critic, an off-policy maximum-entropy actor-critic algorithm that maximises expected reward plus a policy-entropy term, giving stable exploration in continuous action spaces. The environment is a Gym-style simulator whose transition model is an LSTM trained on two years of logged plant data to predict the next state from the current state, the two metal-salt dosing actions, and exogenous variables; it supplies the multi-step state predictions that make delay training feasible. The delay handling is a set of wrappers, inspired by the random-delay Markov decision process literature, that append an action buffer and delay indicators to the observation so the agent can see which action is actually taking effect and how old its state information is.","core_discovery":"The central discovery, as the authors state it, is that explicitly modelling stochastic delays makes a Soft Actor-Critic policy qualitatively better at the phosphorus removal task than either a delay-unaware SAC agent or the PID controller currently logged at the plant. The random-delay agent learned to hold effluent phosphate close to the target with fewer oscillations, and it did so while slightly lowering combined chemical and tax costs compared with PID. The paper is careful to frame this as a simulated result: the comparison runs the same historical exogenous inputs through an LSTM predictor for both the PID actions and the SAC actions, and the PID actions are reconstructed from the historical log rather than obtained from a live controller. A secondary result is that the reward structure matters: with a non-linear penalty that grows steeply above the target concentration, the delay-aware agents trade some chemical cost for much lower regulatory deviation.","pith_inferences":["A natural next step the paper leaves implicit is a closed-loop trial at the plant, since the simulated head-to-head cannot certify how the LSTM behaves on dosing actions the historical PID never took.","The same delay-randomisation recipe could transfer to other slow industrial dosing loops, such as coagulation, pH correction, or chemical precipitation, where stochastic sensor and actuator lag is the main control obstacle.","Because the delay-aware agent outperformed the delay-unaware one, adding random delay during training may act as a form of domain randomisation that regularises the policy; testing that hypothesis would require an ablation where only the delay distribution changes."],"forward_implications":["Delay-unaware SAC agents underperform even the logged PID controller on cost, so any RL deployment at a similar plant should train with explicit delay modelling rather than on an idealised, delay-free environment.","Random-delay training appears to dominate constant-delay training, suggesting that stochastic delay ranges, not just worst-case fixed delays, should be part of the training distribution.","A policy trained on the LSTM simulator can keep effluent phosphate within regulatory limits while using less chemical and paying less phosphorus tax than the existing controller, which is the combination a plant operator actually wants.","The reward function with a steep non-linear penalty is what makes the agent choose compliance over raw cost minimisation, so the reward design is as load-bearing as the algorithm."],"supporting_citations":[{"why":"Supplies the Soft Actor-Critic algorithm that is the paper's learning core.","marker":"[5]"},{"why":"Supplies the delay-wrapper design and the random-delay Markov decision process framing used to train under constant and random delays.","marker":"[10]"},{"why":"Provides the LSTM-based simulator for the phosphorus removal process that the reinforcement learning agents are trained and evaluated on.","marker":"[11]"},{"why":"Supplies the improved multi-step LSTM simulation method used in the environment for delayed feedback training.","marker":"[12]"},{"why":"Supports the multi-step simulation improvement using exogenous state variables, part of the environment's prediction machinery.","marker":"[13]"},{"why":"Documents the plant dataset used to train the LSTM and to reconstruct the historical PID actions for the comparison.","marker":"[23]"}],"fun_headline_variants":["Random-delay SAC agent cuts phosphorus 36% in wastewater sim","Delay-aware RL beats PID on phosphorus and cost in testbed","Soft Actor-Critic with stochastic delays trims phosphorus 36%","AI that learns random delays cuts phosphorus emissions by 36%","Reinforcement learning with delays bests PID on phosphorus and price"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LSTM simulator, trained on historical plant data, predicts the next plant state accurately for the dosing actions the SAC policies choose, including actions outside the range the historical PID controller used, so the head-to-head comparison measures the controllers rather than the simulator's errors.","fun_headline_variants_meta":{"raw":{"variants":["Random-delay SAC agent cuts phosphorus 36% in wastewater sim","Delay-aware RL beats PID on phosphorus and cost in testbed","Soft Actor-Critic with stochastic delays trims phosphorus 36%","AI that learns random delays cuts phosphorus emissions by 36%","Reinforcement learning with delays bests PID on phosphorus and price"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000358,"raw_usage":{"total_tokens":1940,"prompt_tokens":945,"completion_tokens":995,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":906}},"tokens_in":561,"tokens_out":995,"duration_ms":7727,"temperature":1.0,"reasoning_tokens":906,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:19:19.931175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the plant's logged dataset and, for each logged state-action pair, replay the action through the LSTM and compare its multi-step prediction against the state actually recorded at the corresponding future time, focusing on time steps where the dosing action lies in the upper tail of the historical distribution; if the prediction error grows with action novelty, the reported 36 percent emission reduction is not a stable result. A stronger test is a closed-loop trial in which the trained random-delay SAC policy sets the salt doses at the real plant for several weeks and effluent phosphate is compared with the PID baseline over the same inflow conditions.","supporting_citations":[{"cited_title":"Haarnoja, A","cited_arxiv_id":null,"evidence_quote":"Supplies the Soft Actor-Critic algorithm that is the paper's learning core."},{"cited_title":"Bouteiller, S","cited_arxiv_id":null,"evidence_quote":"Supplies the delay-wrapper design and the random-delay Markov decision process framing used to train under constant and random delays."},{"cited_title":"Mohammadi, M","cited_arxiv_id":null,"evidence_quote":"Provides the LSTM-based simulator for the phosphorus removal process that the reinforcement learning agents are trained and evaluated on."},{"cited_title":"Improved Long Short-Term Memory-based Wastewater Treatment Simulators for Deep Reinforcement Learning","cited_arxiv_id":"2403.15091","evidence_quote":"Supplies the improved multi-step LSTM simulation method used in the environment for delayed feedback training."},{"cited_title":"Mohammadi, D","cited_arxiv_id":null,"evidence_quote":"Supports the multi-step simulation improvement using exogenous state variables, part of the environment's prediction machinery."},{"cited_title":"Wastewater Treatment Plant Data for Nutrient Removal System","cited_arxiv_id":"2407.05346","evidence_quote":"Documents the plant dataset used to train the LSTM and to reconstruct the historical PID actions for the comparison."}],"review_version":1}