pith. sign in

arxiv: 2606.32003 · v1 · pith:YMITROGInew · submitted 2026-06-30 · 📡 eess.SY · cs.SY

On the Comparison of Reinforcement Learning and Adaptive Control for Linear Systems under Packet Loss and Uncertainty

Moh Kamalul Wafi This is my paper

Pith reviewed 2026-07-01 03:27 UTC · model grok-4.3

classification 📡 eess.SY cs.SY
keywords adaptive quantized controlreinforcement learningpacket losslinear systemsmodel uncertaintyLyapunov stabilitynetworked control
0
0 comments X

The pith

Adaptive quantized control maintains stability under packet loss and model uncertainty where reinforcement learning degrades.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares adaptive quantized control (AQC) that uses acknowledgment messages to handle packet losses against a deep deterministic policy gradient (DDPG) agent trained only on the nominal model without acknowledgments. Both address uncertain linear systems with input quantization over lossy channels, plus a switch to a more unstable system during operation. In the nominal training environment DDPG shows quicker transients and better damping, yet AQC keeps stability across uncertainty, losses, and switching through its Lyapunov design. The work therefore frames a concrete trade-off between data-driven speed in known conditions and model-based guarantees when conditions change.

Core claim

Under model uncertainty, packet loss, and dynamic switching, the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees, whereas the DDPG controller achieves faster transient responses and improved damping only within its training environment.

What carries the argument

Lyapunov-based stability proof in AQC that incorporates acknowledgment messages to compensate for packet losses while adapting to unknown dynamics.

If this is right

  • AQC can be applied to networked uncertain linear systems where stability must be guaranteed despite losses and changes.
  • DDPG can deliver faster responses when the operating conditions match the training distribution exactly.
  • Designers face an explicit choice between learned performance in matched conditions and provable robustness when conditions vary.
  • Acknowledgment messages enable the adaptive controller to recover from losses in a way unavailable to the RL baseline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Testing other RL methods that receive partial loss information might narrow the robustness gap observed here.
  • The same comparison on nonlinear plants would show whether the Lyapunov advantage persists beyond linear dynamics.
  • Running the experiment with real communication hardware would check whether the numerical packet-loss model matches physical behavior.

Load-bearing premise

Training the DDPG agent solely on the nominal model without acknowledgments and testing only in chosen numerical scenarios gives a fair picture of how each method would behave on physical hardware.

What would settle it

A hardware test in which the DDPG policy keeps the system stable across the same model mismatch, packet-loss rates, and switching events that cause the AQC to lose stability would falsify the reported robustness advantage.

Figures

Figures reproduced from arXiv: 2606.32003 by Moh Kamalul Wafi.

Figure 1
Figure 1. Figure 1: Adaptive quantized control (K, v) method with time-varying quantizer q(k, u(k)). The solid arrows between blocks mean the connected information whereas the dashed arrow show there exists a possibility not being connected. The dashed arrows behind blocks deduces the time-varying. estimation and feedback control are essential for maintaining system performance [14,15]. Note that with stabilization guaranteed… view at source ↗
Figure 2
Figure 2. Figure 2: Left: Logarithmic quantizer q for the scalar case (m = 1). Right: Example of the corresponding sector bounds with M1(k) ≡ 1 and M2(k) ∈ {1 + ˆaµ j i : j ∈ I}. Definition 1. Consider the stochastic discrete-time system xk+1 = f(xk, yk+1), k ∈ N0, (10) where xk ∈ R n and {yk : k ∈ N0} is an R d -valued stochastic process defined on the probability space (Ω, F, P). Here, Ω denotes the sample space, F the asso… view at source ↗
Figure 3
Figure 3. Figure 3: Actor–critic DDPG controller with the time-varying quantizer q(k, u(k)). The arrows have the same interpretation as those in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Architecture of the actor–critic DDPG controller. The activation function ∆i is applied elementwise according to ∆i(vi) = λ(v1), . . . , λ(vmi ) ⊤ , where λ(·) denotes either the hyperbolic tangent or the ReLU activation function. The output layer has dimension nu, matching the dimension of the control input. For subsequent analysis, define vq = [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performances of AQC and DRL with two different initial condition x0 and various dynamic change time td. Defining the state variables x1(k) = z(k) and x2(k) = z(k + 1). The nominal and perturbed system matrices are A =  0 1 −α1 −β1  , Aw =  0 1 −α2 −β2  , with x = [x1, x2] ⊤, x(0) = x0, and B = [0, bζ] ⊤, ∀ζ. For the AQC design, we choose R = I2, which yields the Riccati solution P = diag(1, 2) > I2. Th… view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of AQC and DRL under varying levels of packet loss η(k). α2 = 1 2 3 4 5 . . . 18 19 20 RL X X × × × . . . × × × AQC X X X X X . . . X × × AC X X X X X . . . X X X [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
read the original abstract

This paper presents a comparative study between Adaptive Quantized Control (AQC) and Deep Deterministic Policy Gradient (DDPG) reinforcement learning for uncertain linear systems with input quantization over communication channels subject to packet loss. The considered setting also includes dynamic switching from a nominal unstable system to a more unstable one during operation. The AQC is designed for unknown system dynamics using acknowledgment messages to compensate for packet losses, whereas the DDPG controller is trained using the nominal system model without acknowledgment messages. Numerical results show that the DDPG controller achieves faster transient responses and improved damping within its training environment. However, under model uncertainty, packet loss, and dynamic switching, the AQC consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees. These results highlight the trade-off between data-driven performance and model-based robustness, and provide insight into the applicability of reinforcement learning and adaptive control for networked uncertain systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper compares Adaptive Quantized Control (AQC), designed with acknowledgment messages for packet-loss compensation, against a DDPG reinforcement-learning controller trained on the nominal model without acknowledgments, for uncertain linear systems subject to input quantization, packet loss, and dynamic switching between unstable models. Numerical results indicate superior transient performance for DDPG in the training environment but consistently better robustness for AQC under uncertainty and packet loss, which the abstract attributes to Lyapunov stability guarantees.

Significance. A controlled comparison of model-based adaptive control with explicit stability analysis versus data-driven RL in networked settings with quantization and switching would be useful if the controllers can be placed on equal footing; the presence of Lyapunov guarantees is a methodological strength when the experimental design isolates their contribution.

major comments (2)
  1. [Abstract] Abstract: the claim that AQC 'consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees' is not isolated from the explicit design difference that only AQC receives acknowledgment messages while 'the DDPG controller is trained using the nominal system model without acknowledgment messages.' Any performance gap under packet loss is therefore confounded by unequal information access rather than attributable solely to the stability analysis.
  2. [Numerical results] Numerical results section: the manuscript supplies no information on simulation parameters, number of independent trials, statistical tests, or the precise metric used to quantify post-training generalization, leaving the central robustness claim weakly supported by the reported experiments.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major comment below and indicate the corresponding revisions to the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that AQC 'consistently demonstrates superior robustness owing to its rigorous Lyapunov stability guarantees' is not isolated from the explicit design difference that only AQC receives acknowledgment messages while 'the DDPG controller is trained using the nominal system model without acknowledgment messages.' Any performance gap under packet loss is therefore confounded by unequal information access rather than attributable solely to the stability analysis.

    Authors: We agree that the abstract's phrasing risks overstating the role of the Lyapunov analysis in isolation. The performance difference arises from the full AQC design, which integrates adaptive laws (supported by Lyapunov guarantees) with acknowledgment-based packet-loss compensation; the DDPG controller, by contrast, is deliberately trained without acknowledgments on the nominal model. This reflects typical usage of each approach rather than an attempt to isolate the stability analysis alone. To prevent misinterpretation, we will revise the abstract to state that AQC's robustness follows from its integrated design, including both the Lyapunov-based adaptive component and the acknowledgment mechanism. revision: yes

  2. Referee: [Numerical results] Numerical results section: the manuscript supplies no information on simulation parameters, number of independent trials, statistical tests, or the precise metric used to quantify post-training generalization, leaving the central robustness claim weakly supported by the reported experiments.

    Authors: We acknowledge the omission of these experimental details. In the revised version we will add: (i) all simulation parameters (system matrices, quantization levels, packet-loss probabilities, switching instants, and DDPG training hyperparameters); (ii) the number of independent Monte-Carlo trials performed; (iii) the performance metrics employed (settling time, peak overshoot, and integral of squared error); and (iv) a statement that results are reported as averages without formal statistical hypothesis testing. These additions will make the robustness comparison reproducible and better supported. revision: yes

Circularity Check

0 steps flagged

No significant circularity; comparison is self-contained

full rationale

The paper performs an empirical comparison of two independently designed controllers: AQC using Lyapunov-based adaptive design with ACKs for packet loss, and DDPG trained on the nominal model without ACKs. No equations, predictions, or central claims reduce by construction to fitted inputs, self-citations, or renamed known results. The attribution of robustness to Lyapunov guarantees follows from the separate stability analysis and numerical evaluation; the designs do not feed into each other. Any information asymmetry in the baseline is a potential experimental confound rather than a circular reduction in the derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract alone supplies no explicit free parameters, axioms, or invented entities; full assessment requires the manuscript.

pith-pipeline@v0.9.1-grok · 5682 in / 1041 out tokens · 53194 ms · 2026-07-01T03:27:12.936062+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · 2 internal anchors

  1. [1]

    Deep reinfo rcement learning for multiagent systems: A review of challenges, solutions, and applications,

    T. T. Nguyen, N. D. Nguyen, and S. Nahavandi, “Deep reinfo rcement learning for multiagent systems: A review of challenges, solutions, and applications,” IEEE Transactions on Cybernetics , vol. 50, no. 9, pp. 3826–3839, 2020

    work page 2020

  2. [2]

    A com prehensive survey on graph neural networks,

    Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P . S. Y u, “A com prehensive survey on graph neural networks,” IEEE Transactions on Neural Networks and Learning Systems , vol. 32, no. 1, pp. 4–24, 2021

    work page 2021

  3. [3]

    Deep reinforcement learning: A survey,

    X. Wang, S. Wang, X. Liang, D. Zhao, J. Huang, X. Xu, B. Dai, and Q. Miao, “Deep reinforcement learning: A survey,” IEEE Transactions on Neural Networks and Learning Systems , pp. 1–15, 2022

    work page 2022

  4. [4]

    Deep reinforcement learning control of a boiling water reactor,

    X. Chen and A. Ray, “Deep reinforcement learning control of a boiling water reactor,” IEEE Transactions on Nuclear Science, vol. 69, no. 8, pp. 1820–1832, 2022

    work page 2022

  5. [5]

    Deep reinforcement learnin g-based automatic exploration for navigation in unknown environment,

    H. Li, Q. Zhang, and D. Zhao, “Deep reinforcement learnin g-based automatic exploration for navigation in unknown environment,” IEEE Transactions on Neural Networks and Learning Systems , vol. 31, no. 6, pp. 2064– 2076, 2020

    work page 2064

  6. [6]

    Investigating the ef fectiveness of reinforcement learning in closed-loop systems with time delays,

    M. K. Wafi, M. Siami, and M. Sznaier, “Investigating the ef fectiveness of reinforcement learning in closed-loop systems with time delays,” in 2024 American Control Conference (ACC), pp. 4149–4154, 2024

    work page 2024

  7. [7]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. V eness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529– 533, Feb 2015

    work page 2015

  8. [8]

    Continuous control with deep reinforcement learning,

    T. P . Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” 2015

    work page 2015

  9. [9]

    Stabilization of linear systems w ith limited information,

    N. Elia and S. Mitter, “Stabilization of linear systems w ith limited information,” IEEE Transactions on Automatic Control, vol. 46, no. 9, pp. 1384–1400, 2001

    work page 2001

  10. [10]

    The sector bound approach to quantized feedback control,

    M. Fu and L. Xie, “The sector bound approach to quantized feedback control,” IEEE Transactions on Automatic Control, vol. 50, no. 11, pp. 1698–1711, 2005

    work page 2005

  11. [11]

    Adaptive quanti zed control for linear uncertain discrete-time systems,

    T. Hayakawa, H. Ishii, and K. Tsumura, “Adaptive quanti zed control for linear uncertain discrete-time systems,” Automatica, vol. 45, no. 3, pp. 692–700, 2009

    work page 2009

  12. [12]

    Adapti ve quantized control for linear uncertain systems over channels subject to packet loss,

    M. Siami, T. Hayakawa, H. Ishii, and K. Tsumura, “Adapti ve quantized control for linear uncertain systems over channels subject to packet loss,” in 49th IEEE Conference on Decision and Control (CDC), pp. 4655–4660, 2010

    work page 2010

  13. [13]

    Adaptive quanti zed control for nonlinear uncertain systems,

    T. Hayakawa, H. Ishii, and K. Tsumura, “Adaptive quanti zed control for nonlinear uncertain systems,” Systems & Control Letters, vol. 58, no. 9, pp. 625–632, 2009

    work page 2009

  14. [14]

    Distributed estimatio n with decentralized control for quadruple-tank process,

    M. K. Wafi and B. L. Widjiantoro, “Distributed estimatio n with decentralized control for quadruple-tank process,” arXiv preprint arXiv:2304.04763 , 2025

    work page arXiv 2025

  15. [15]

    Adaptive Covariance Kalman Filtering and Nonlinear Decoupling Control via Feedback Linearization for a Three-Tank Process

    B. L. Widjiantoro, K. Indriawati, and M. K. Wafi, “Adapti ve covariance kalman filtering and nonlinear decoupling control via feedback linearization for a three-tank proces s,” arXiv preprint arXiv:2304.04144 , 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  16. [16]

    The role of s ystems theory in control oriented learning,

    M. Sznaier, A. Olshevsky, and E. D. Sontag, “The role of s ystems theory in control oriented learning,” in 25th International Symposium on Mathematical Theory of Network s and Systems , 2022

    work page 2022

  17. [17]

    Deterministic policy gradient algo- rithms,

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, an d M. Riedmiller, “Deterministic policy gradient algo- rithms,” in Proceedings of the 31st International Conference on Machin e Learning (E. P . Xing and T. Jebara, eds.), vol. 32 of Proceedings of Machine Learning Research , (Bejing, China), pp. 387–395, PMLR, 22–24 Jun 2014. 12

    work page 2014

  18. [19]

    A comparative analysis of reinfo rcement learning and adaptive control techniques for linear uncertain systems,

    M. K. Wafi and M. Siami, “A comparative analysis of reinfo rcement learning and adaptive control techniques for linear uncertain systems,” in 2023 Proceedings of the Conference on Control and its Applications (CT), pp. 25–32, 2023

    work page 2023

  19. [20]

    Lyapunov criterio n for stochastic systems and its applications in distributed computation,

    Y . Qin, M. Cao, and B. D. O. Anderson, “Lyapunov criterio n for stochastic systems and its applications in distributed computation,” IEEE Transactions on Automatic Control, vol. 65, no. 2, pp. 546–560, 2020

    work page 2020

  20. [21]

    A partial history of the early developme nt of continuous-time nonlinear stochastic systems the- ory,

    H. J. Kushner, “A partial history of the early developme nt of continuous-time nonlinear stochastic systems the- ory,” Automatica, vol. 50, no. 2, pp. 303–334, 2014

    work page 2014

  21. [22]

    Stability analysis usi ng quadratic constraints for systems with neural network controllers,

    H. Yin, P . Seiler, and M. Arcak, “Stability analysis usi ng quadratic constraints for systems with neural network controllers,” IEEE Transactions on Automatic Control, vol. 67, no. 4, pp. 1980–1987, 2022

    work page 1980

  22. [23]

    Model r eference adaptive control of networked systems with state and input delays,

    M. K. Wafi, K. Indriawati, and B. L. Widjiantoro, “Model r eference adaptive control of networked systems with state and input delays,” International Journal of Electrical and Computer Engineer ing (IJECE), vol. 14, p. 5055, Oct. 2024

    work page 2024

  23. [24]

    Distributed adaptive control of disturbed interconnected systems with high-order tuners,

    M. K. Wafi and M. Siami, “Distributed adaptive control of disturbed interconnected systems with high-order tuners,” IEEE Control Systems Letters, vol. 8, pp. 1421–1426, 2024

    work page 2024

  24. [25]

    Cooperative $\mathcal{H}_\infty$ Fault-Tolerant Tracking with ISS Guarantees for Networked Systems with Sensor Faults

    M. K. Wafi, Y . E. Nugraha, B. L. Widjiantoro, and K. Indria wati, “CooperativeH∞ fault-tolerant tracking with iss guarantees for networked systems with sensor faults,” arXiv preprint arXiv:2604.03921 , 2026. 13

    work page internal anchor Pith review Pith/arXiv arXiv 2026