Pith. sign in

REVIEW 2 major objections 5 minor 40 references

A deep-RL agent can raise torque on a robot joint by ~54% while staying below anomaly detectors in a digital-twin control loop.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 08:55 UTC pith:LFYS4T7B

load-bearing objection Solid first DRL wear-out attack on DT-controlled robots; the sim claim holds, the lifespan numbers are just illustration. the 2 major comments →

arxiv 2607.10830 v1 pith:LFYS4T7B submitted 2026-07-12 cs.CR

Automated Stealthy Wear-Out Attack on Digital Twins With Deep Reinforcement Learning

Joshua Haworth , Aryan Pasikhani , George Pavlides , Prosanta Gope , John Clark This is my paper
classification cs.CR
keywords Digital TwinsDeep Reinforcement Learningwear-out attackSoft Actor-Criticanomaly detectionUR10eindustrial control systemsadversarial machine learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Digital twins send real-time control commands to physical machines and thereby open a new attack surface. This paper shows that a deep reinforcement learning agent, sitting on the path between the twin and the machine, can learn small, continuous changes to those commands that steadily increase mechanical stress on one chosen joint of a UR10e arm. The agent is trained to maximise torque while keeping an ensemble of autoencoder anomaly detectors below their alarm threshold. Soft Actor-Critic proved the most sample-efficient and stable of four algorithms tested. In simulation the mean torque on Wrist 3 rose from 33 Nm to 51 Nm (about 54 percent) while the average anomaly score stayed near process noise, so the attack remains undetected. The result is offered as evidence that intelligent, low-and-slow wear-out is now practical against twin-driven industrial robots and that existing reconstruction-based detectors are insufficient.

Core claim

A grey-box Soft Actor-Critic adversary that observes only joint positions and can add offsets of at most 0.01 m to waypoints can raise mean torque on the UR10e Wrist 3 joint by approximately 54 percent over 1000 episodes while producing an average anomaly probability of roughly 21 percent—statistically indistinguishable from normal process noise and well below the 50 percent detection threshold—thereby accelerating simulated mechanical wear without triggering the twin’s ensemble autoencoder detectors.

What carries the argument

The SAC policy trained inside a MuJoCo Gymnasium environment whose reward is torque on the target joint times (1 − anomaly probability), with unreachable or safety-limit-violating actions penalised by −1; the resulting low-and-slow waypoint perturbations are the attack carrier.

Load-bearing premise

That the torque numbers produced by the MuJoCo model under the chosen inverse-kinematics tolerance, combined with a generic aluminium fatigue-exponent range, correctly predict real-world reduction in the lifespan of a UR10e Wrist 3 joint.

What would settle it

Mount a physical UR10e, run the identical SAC policy for several hundred hours while logging true joint torque and temperature, and check whether measured wear or remaining useful life matches the 454–2583-hour extrapolation derived from the simulated 54 percent torque increase.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents a grey-box DRL adversary that sits on the VE–PE control path of a DT-enabled robotic system and applies small per-axis waypoint offsets to raise mechanical torque on a chosen joint while remaining below the detection threshold of an ensemble of autoencoder-based anomaly detectors. Using a MuJoCo model of the UR10e, the authors benchmark SAC, TD3, PPO and A2C, select SAC, ablate entropy coefficient, action-range and observation subsets, and report that the trained policy raises mean Wrist-3 torque from 33.12 Nm to 51.15 Nm (~54 %) over 1000 episodes while producing a mean anomaly probability of ~20.6 % (baseline process noise ~17.9 %). They further compare against constant and random baselines, examine limited transfer to UR5e and Franka Panda, release code and policies, and discuss dual-use considerations.

Significance. If the simulation results hold under the stated threat model, the work supplies the first concrete demonstration that a sample-efficient off-policy DRL agent can synthesise a persistent, low-and-slow wear-out attack against a DT-mediated industrial robot while systematically exploiting reconstruction-based detectors. The public release of environment, trained policies and IDS models is a clear strength that enables subsequent defensive research. The systematic algorithm comparison, grey-box observation ablations and naïve-attack baselines raise the technical bar relative to the manually tuned FDIA literature on DTs. The fatigue-to-lifespan extrapolation is secondary; the core technical contribution is the measured torque elevation under stealth constraints.

major comments (2)
  1. [§5.5, Eqs. (10)–(13)] §5.5, Eqs. (10)–(13) and the abstract/conclusions: the quantitative lifespan reduction (454–2583 h) is obtained by feeding MuJoCo torque ratios into a literature aluminium fatigue exponent range k∈[6,10] with no manufacturer fatigue curve or physical wear data for UR10e Wrist 3. The calculation is post-hoc and does not affect the measured torque or anomaly scores, yet the abstract and §7 present “accelerated degradation and increased maintenance costs” as established outcomes. Either supply a calibrated fatigue model or reframe the numbers as purely illustrative bounds and remove the specific hour figures from the abstract and conclusions.
  2. [Abstract, §1, §4.1] Abstract and §1 claim evaluation “in an industrial setting using the UR10e robotic arm,” while every quantitative result in §5 is obtained inside MuJoCo with a fixed three-waypoint task and a chosen IK tolerance (§4.1, §6.2). The discussion correctly notes the sim-to-real gap, but the abstract and contribution statements currently overstate physical fidelity. A precise scoping sentence (simulation only; no hardware transfer) is required so that the central claim remains accurately bounded.
minor comments (5)
  1. [§5.2, §5.5] §5.2 takeaway states that entropy coefficient 0.3 is used for subsequent experiments, yet §5.5 reports results with the “auto” coefficient. Align the text and, if both were run, report which configuration produced the 51.15 Nm figure.
  2. [§5.3] Figure numbering for the action-space plots is inconsistent (text refers to Fig. 6a/b while captions appear as separate “Figure 4/5” and later “Figure 6”). Renumber for sequential consistency.
  3. [§4.3, Table 2] Table 2 lists five detectors (including a plain LSTM) while the ensemble description in §4.3 mentions four autoencoders; clarify membership of the ensemble used for P_anom.
  4. [Appendix C] Appendix C reports perfect or near-perfect F1/AUC for several detectors on the three-waypoint task. A short remark on whether this indicates task simplicity or detector over-fit would help readers judge the difficulty of the stealth objective.
  5. [Throughout] Minor typos: “Denail of Service”, “a.k.a grey box”, “low & slow” inconsistently hyphenated, and “T ¨arneberg” spacing. A proof-reading pass is sufficient.

Circularity Check

0 steps flagged

No significant circularity: torque elevation and anomaly scores are direct MuJoCo/IDS measurements of a trained SAC policy; fatigue reduction is a post-hoc literature formula application, not a fitted or self-defined prediction.

full rationale

The paper's central results (SAC raising mean Wrist-3 torque from 33.12 Nm to 51.15 Nm while keeping mean anomaly probability ≈20.63 % < 50 % threshold) are obtained by training an agent whose reward is explicitly R = τ_target · (1 − P_anom) (Eq. 7) and then measuring the independent environment outputs τ_target (from mjData) and P_anom (from the ensemble autoencoders) under the learned policy. This is ordinary RL evaluation, not algebraic rearrangement of fitted constants. Algorithm benchmarking, action/observation ablations, and transfer experiments likewise report empirical returns and torques. The lifespan claim (Eqs. 10–13) simply multiplies the measured torque ratio by a literature aluminium fatigue exponent k ∈ [6,10] and a brochure L_normal = 35 000 h; it is presented as an illustrative consequence, not as a first-principles derivation or a quantity the policy was trained to predict. No self-citation supplies a uniqueness theorem, ansatz, or load-bearing premise; standard RL citations (SAC, TD3, PPO, A2C) and DT/attack literature are external. The simulation supplies both training signal and evaluation metrics—standard practice and not circular by the stated criteria. Artefacts are released, rendering the simulation claim externally checkable. Hence score 0 with empty steps.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claim rests on a grey-box network adversary, a physics simulator treated as ground truth, torque-as-wear proxy, live anomaly-score feedback, and literature fatigue exponents. No new physical entities are postulated; free parameters are the usual RL and attack-design knobs plus the fatigue-range bounds used for the lifespan headline.

free parameters (5)
  • waypoint modification range = [−0.01, 0.01] (final)
    Initially set to 10 % of arm reach [−0.1,0.1], later ablated to [−0.01,0.01] as the preferred stealth/impact trade-off; chosen by experiment rather than derived.
  • SAC entropy regularisation coefficient = 0.3
    Swept over {0.1,0.3,0.5,0.7,0.9,auto}; 0.3 selected for subsequent runs.
  • material fatigue exponent k = 6–10
    Taken from aluminium literature range 6–10 to convert torque ratio into lifespan reduction factor; not measured on UR10e Wrist 3.
  • anomaly detection threshold for 'detectable' = 0.5
    Agent declared undetectable if P_anom ≤ 50 %; threshold chosen by authors for the stealth experiment.
  • learning rates and buffer sizes (Table 1) = SAC lr=3e-4, buffer=1e6, etc.
    Standard Stable-Baselines3 defaults / common values; affect convergence but not the form of the claim.
axioms (6)
  • domain assumption Torque measured in MuJoCo is a sufficient proxy for mechanical wear of the target joint.
    Stated in §4.1 with citation to joint-wear literature; no temperature/vibration/wear-particle validation.
  • domain assumption Adversary can observe live anomaly-detector scores (and therefore use them in the reward) via L2 access.
    Explicit in §3.1.2 and reward Eq. 7; without this the stealth term cannot be optimised online.
  • domain assumption The MuJoCo Menagerie UR10e model plus Robotics Toolbox IK produces dynamics faithful enough for wear-out conclusions.
    §4.1; authors note real-world humidity, temperature and bearing fatigue lie outside scope (§6.2).
  • domain assumption Dolev–Yao adversary on the VE–PE path can inject arbitrary waypoint offsets within the chosen range.
    Threat model §3; assumes ICS protocols (e.g. MODBUS-TCP) remain unauthenticated.
  • domain assumption Grey-box observation of (at least) target-joint position is available to the agent.
    §3 Observation; ablations show even single-joint position suffices.
  • standard math Standard continuous-control RL convergence assumptions (Markov / POMDP, discounted return maximisation).
    POMDP tuple and SAC updates §3.1.

pith-pipeline@v1.1.0-grok45 · 27364 in / 3452 out tokens · 59880 ms · 2026-07-14T08:55:43.667684+00:00 · methodology

0 comments
read the original abstract

Digital Twins (DTs) have emerged as pivotal enablers of Industry 4.0, offering transformative capabilities such as real-time monitoring, advanced simulation, and precise control of physical assets. By bridging the physical and virtual domains, DTs facilitate seamless integration of data-driven decision-making and operational optimisation. However, this seamless interaction significantly expands the attack surface of industrial systems, creating vulnerabilities that adversaries can exploit. This paper introduces a novel and stealthy wear-out attack leveraging Deep Reinforcement Learning (DRL) to target DT-enabled infrastructures. The adversary strategically and covertly manipulates control signals, inducing increased torque on a specific joint to accelerate wear and tear while evading detection by a state-of-the-art anomaly detection system. Extensive benchmarking of reinforcement learning algorithms - including Twin Delayed Deep Deterministic Policy Gradient (TD3), Soft Actor-Critic (SAC), Proximal Policy Optimisation (PPO), and Advantage Actor-Critic (A2C) - revealed that SAC consistently outperformed its counterparts in terms of sample efficiency, stability, and overall attack effectiveness. We evaluate the proposed adversary in an industrial setting using the UR10e robotic arm. Results demonstrate the adversary's ability to significantly elevate torque levels on the targeted joint, leading to accelerated degradation and increased maintenance costs, all while operating stealthily and avoiding detection. Our findings highlight the substantial risks posed by DRL-driven adversaries to DT-enabled environments and emphasise the critical need for robust defence mechanisms to protect critical industrial systems.

Figures

Figures reproduced from arXiv: 2607.10830 by Aryan Pasikhani, George Pavlides, John Clark, Joshua Haworth, Prosanta Gope.

Figure 1
Figure 1. Figure 1: DRL-assisted adversary targeting an industrial robotic system by exploiting vulnerabilities in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Algorithm Benchmarking performance during [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Entropy Regularisation Coefficient comparison [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Action Space Experimentation - Training Re [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Action Space Experimentation - Testing Re [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Action Space Experimentation. recall of 1.0000 with precision below 0.97) indicates that the model captures malicious sequences in the evaluation set but produces a small number of false positives on normal trajectories. For an IDS deployed in an industrial setting where false alarms incur operational cost, this is the more tolerable failure mode of the two, missed detections would enable the attacks studi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 1 canonical work pages

  1. [1]

    Industry 4.0, digitization, and opportunities for sustainability,

    M. Ghobakhloo, “Industry 4.0, digitization, and opportunities for sustainability,”Journal of Cleaner Production, vol. 252, p. 119869, 2020. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S0959652619347390

    2020

  2. [2]

    Digital twin in indus- try: State-of-the-art,

    F. Tao, H. Zhang, A. Liu, and A. Y . C. Nee, “Digital twin in indus- try: State-of-the-art,”IEEE Transactions on Industrial Informatics, vol. 15, no. 4, pp. 2405–2415, 2019

    2019

  3. [3]

    Digital twin driven prognos- tics and health management for complex equipment,

    F. Tao, M. Zhang, Y . Liu, and A. Nee, “Digital twin driven prognos- tics and health management for complex equipment,”CIRP Annals, vol. 67, no. 1, pp. 169–172, 2018. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S0007850618300799

    2018

  4. [4]

    Elegant: Security of critical infrastructures with digital twins,

    B. Sousa, M. Arieiro, V . Pereira, J. Correia, N. Lourenc ¸o, and T. Cruz, “Elegant: Security of critical infrastructures with digital twins,”IEEE Access, vol. 9, pp. 107 574–107 588, 2021

    2021

  5. [5]

    Prototyping intrusion detection in an industrial cloud-native digital twin,

    W. T ¨arneberg, P. Skarin, C. Gehrmann, and M. Kihl, “Prototyping intrusion detection in an industrial cloud-native digital twin,” in 2021 22nd IEEE International Conference on Industrial Technol- ogy (ICIT), vol. 1, 2021, pp. 749–755

    2021

  6. [6]

    Assessment of scada system vulnerabil- ities,

    G. Yadav and K. Paul, “Assessment of scada system vulnerabil- ities,” in2019 24th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), 2019, pp. 1737– 1744

    2019

  7. [7]

    A survey on privacy attacks against digital twin systems in ai-robotics,

    I. A. Fernandez, S. Neupane, T. Chakraborty, S. Mitra, S. Mittal, N. Pillai, J. Chen, and S. Rahimi, “A survey on privacy attacks against digital twin systems in ai-robotics,” 2024. [Online]. Available: https://arxiv.org/abs/2406.18812

    Pith/arXiv arXiv 2024

  8. [9]

    A secu- rity framework in digital twins for cloud-based industrial control systems: Intrusion detection and mitigation,

    F. Akbarian, W. T ¨arneberg, E. Fitzgerald, and M. Kihl, “A secu- rity framework in digital twins for cloud-based industrial control systems: Intrusion detection and mitigation,” in2021 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA ), 2021, pp. 01–08

    2021

  9. [11]

    Intrusion detection in digital twins for industrial control systems,

    F. Akbarian, E. Fitzgerald, and M. Kihl, “Intrusion detection in digital twins for industrial control systems,” in2020 International Conference on Software, Telecommunications and Computer Net- works (SoftCOM), 2020, pp. 1–6

    2020

  10. [12]

    Digital twin-based cyber-attack detection framework for cyber- physical manufacturing systems,

    E. C. Balta, M. Pease, J. Moyne, K. Barton, and D. M. Tilbury, “Digital twin-based cyber-attack detection framework for cyber- physical manufacturing systems,”IEEE Transactions on Automa- tion Science and Engineering, vol. 21, no. 2, pp. 1695–1712, 2024

    2024

  11. [13]

    Availability attacks on computing systems through alteration of environmental control: smart malware approach,

    K. Chung, Z. T. Kalbarczyk, and R. K. Iyer, “Availability attacks on computing systems through alteration of environmental control: smart malware approach,” inProceedings of the 10th ACM/IEEE International Conference on Cyber-Physical Systems, ser. ICCPS ’19. New York, NY , USA: Association for Computing Machinery, 2019, p. 1–12. [Online]. Available: http...

    work page doi:10.1145/3302509.3311041 2019

  12. [14]

    Generating adversarial examples against machine learning-based intrusion detector in industrial control systems,

    J. Chen, X. Gao, R. Deng, Y . He, C. Fang, and P. Cheng, “Generating adversarial examples against machine learning-based intrusion detector in industrial control systems,”IEEE Transactions on Dependable and Secure Computing, vol. 19, no. 3, pp. 1810– 1825, 2022

    2022

  13. [15]

    On the use of reinforcement learning for attacking and defending load frequency control,

    A. S. Mohamed and D. Kundur, “On the use of reinforcement learning for attacking and defending load frequency control,” IEEE Transactions on Smart Grid, vol. 15, no. 3, p. 3262–3277, May 2024. [Online]. Available: http://dx.doi.org/10.1109/TSG. 2023.3343100

    doi:10.1109/tsg 2024

  14. [16]

    A reinforcement learning approach to undetectable attacks against automatic generation con- trol,

    E. Shereen, K. Kazari, and G. D ´an, “A reinforcement learning approach to undetectable attacks against automatic generation con- trol,”IEEE Transactions on Smart Grid, vol. 15, no. 1, pp. 959– 972, 2024

    2024

  15. [17]

    Targeted attack synthesis for smart grid vulnerability analysis,

    S. Maiti, A. Balabhaskara, S. Adhikary, I. Koley, and S. Dey, “Targeted attack synthesis for smart grid vulnerability analysis,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 2576–2590. [Online]. Available: https://doi.org/10.1145/35...

    doi:10.1145/3576915.3623155 2023

  16. [18]

    Learning to attack federated learning: A model-based reinforcement learning attack framework,

    H. Li, X. Sun, and Z. Zheng, “Learning to attack federated learning: A model-based reinforcement learning attack framework,” Advances in Neural Information Processing Systems, vol. 35, pp. 35 007–35 020, 2022

    2022

  17. [19]

    Stuxnet: Dissecting a cyberwarfare weapon,

    R. Langner, “Stuxnet: Dissecting a cyberwarfare weapon,”IEEE Security & Privacy, vol. 9, no. 3, pp. 49–51, 2011

    2011

  18. [20]

    A specification-based state replication approach for digital twins,

    M. Eckhart and A. Ekelhart, “A specification-based state replication approach for digital twins,” inProceedings of the 2018 Workshop on Cyber-Physical Systems Security and PrivaCy, ser. CPS-SPC ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 36–47. [Online]. Available: https://doi.org/10.1145/3264888.3264892

    doi:10.1145/3264888.3264892 2018

  19. [21]

    Employ- ing digital twins for security-by-design system testing,

    M. Dietz, L. Hageman, C. von Hornung, and G. Pernul, “Employ- ing digital twins for security-by-design system testing,” inPro- ceedings of the 2022 ACM Workshop on Secure and Trustworthy Cyber-Physical Systems, 2022, pp. 97–106

    2022

  20. [22]

    On the security of public key protocols,

    D. Dolev and A. Yao, “On the security of public key protocols,” IEEE Transactions on Information Theory, vol. 29, no. 2, pp. 198– 208, 1983

    1983

  21. [23]

    Digital twin-based intrusion detection for indus- trial control systems,

    S. A. Varghese, A. D. Ghadim, A. Balador, Z. Alimadadi, and P. Papadimitratos, “Digital twin-based intrusion detection for indus- trial control systems,” in2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). IEEE, 2022, pp. 611–617

    2022

  22. [24]

    A smart digital twin enabled security framework for vehicle-to- grid cyber-physical systems,

    M. Ali, G. Kaddoum, W.-T. Li, C. Yuen, M. Tariq, and H. V . Poor, “A smart digital twin enabled security framework for vehicle-to- grid cyber-physical systems,”IEEE Transactions on Information Forensics and Security, vol. 18, pp. 5258–5271, 2023

    2023

  23. [25]

    Q-learning-based vulner- ability analysis of smart grid against sequential topology attacks,

    J. Yan, H. He, X. Zhong, and Y . Tang, “Q-learning-based vulner- ability analysis of smart grid against sequential topology attacks,” IEEE Transactions on Information Forensics and Security, vol. 12, no. 1, pp. 200–210, 2017

    2017

  24. [26]

    A reinforcement learning approach for sequential decision-making process of attacks in smart grid,

    Z. Ni, S. Paul, X. Zhong, and Q. Wei, “A reinforcement learning approach for sequential decision-making process of attacks in smart grid,” in2017 IEEE Symposium Series on Computational Intelligence (SSCI), 2017, pp. 1–8

    2017

  25. [27]

    Coordinated topology attacks in smart grid using deep reinforcement learning,

    Z. Wang, H. He, Z. Wan, and Y . Sun, “Coordinated topology attacks in smart grid using deep reinforcement learning,”IEEE Transactions on Industrial Informatics, vol. 17, no. 2, pp. 1407– 1415, 2021

    2021

  26. [28]

    Evaluation of re- inforcement learning-based false data injection attack to automatic voltage control,

    Y . Chen, S. Huang, F. Liu, Z. Wang, and X. Sun, “Evaluation of re- inforcement learning-based false data injection attack to automatic voltage control,”IEEE Transactions on Smart Grid, vol. 10, no. 2, pp. 2158–2169, 2019

    2019

  27. [29]

    Vulnerability identification and remediation of fdi attacks in islanded dc microgrids using multiagent reinforcement learning,

    A. J. Abianeh, Y . Wan, F. Ferdowsi, N. Mijatovic, and T. Dragiˇcevi´c, “Vulnerability identification and remediation of fdi attacks in islanded dc microgrids using multiagent reinforcement learning,”IEEE Transactions on Power Electronics, vol. 37, no. 6, pp. 6359–6370, 2022

    2022

  28. [30]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” inInternational conference on machine learning. Pmlr, 2018, pp. 1861–1870

    2018

  29. [31]

    Method for robot manipulator joint wear reduction by finding the optimal robot placement in a robotic cell,

    T. Kot, Z. Bobovsk ´y, A. Vysock ´y, V . Krys, J. ˇSafaˇr´ık, and R. Ru ˇzarovsk´y, “Method for robot manipulator joint wear reduction by finding the optimal robot placement in a robotic cell,”Applied Sciences, vol. 11, no. 12, 2021. [Online]. Available: https://www.mdpi.com/2076-3417/11/12/5398

    2021

  30. [32]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” 2018. [Online]. Available: https://arxiv.org/abs/1801.01290

    Pith/arXiv arXiv 2018

  31. [33]

    Addressing function approx- imation error in actor-critic methods,

    S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approx- imation error in actor-critic methods,” inInternational conference on machine learning. PMLR, 2018, pp. 1587–1596

    2018

  32. [34]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347

    Pith/arXiv arXiv 2017

  33. [35]

    Asynchronous methods for deep reinforcement learning,

    V . Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” 2016. [Online]. Available: https://arxiv.org/abs/1602.01783

    Pith/arXiv arXiv 2016

  34. [36]

    R. S. Sutton and A. G. Barto,Reinforcement learning: An intro- duction. MIT press, 2018

    2018

  35. [37]

    Exploration versus exploitation in reinforcement learning: a stochastic control approach,

    H. Wang, T. Zariphopoulou, and X. Zhou, “Exploration versus exploitation in reinforcement learning: a stochastic control approach,” 2019. [Online]. Available: https://arxiv.org/abs/1812. 01552

    2019

  36. [38]

    Soft Actor Critic,

    OpenAI Spinning Up, “Soft Actor Critic,” https://spinningup. openai.com/en/latest/algorithms/sac.html, 2018, accessed: 2025- 03-21

    2018

  37. [39]

    Fatigue of structures and materials,

    S. Japp, “Fatigue of structures and materials,”Netherlands: Kluwer academic publisher, 2014

    2014

  38. [40]

    Stephens, A

    R. Stephens, A. Fatemi, R. Stephens, and H. Fuchs,Metal Fatigue in Engineering, ser. A Wiley-Interscience publication. Wiley,

  39. [41]

    Available: https://books.google.co.uk/books?id= B2aAPVa1TloC

    [Online]. Available: https://books.google.co.uk/books?id= B2aAPVa1TloC

  40. [42]

    Available: https://www.universal-robots.com/blog/ determine-roi-for-your-palletizing-cobot-application/ Appendix A

    [Online]. Available: https://www.universal-robots.com/blog/ determine-roi-for-your-palletizing-cobot-application/ Appendix A. Acknowledgements This work was supported in part by the Engineering and Physical Sciences Research Council (EPSRC) under Award EP/V039156/1. Appendix B. Ethical Discussion This work introduces an AI-assisted methodology for generat...