Optimal Scheduling for Discounted Age Penalty Minimization in Multi-Loop Networked Control

Mikhail Vilgelm; Onur Ayan; Wolfgang Kellerer

REVIEW 3 major objections 6 minor 16 references

Optimal Scheduling for Discounted Age Penalty Minimization in Multi-Loop Networked Control

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A scheduling policy built on age-of-information states minimizes discounted control error across multiple wireless control loops, outperforming both age-optimal and greedy error baselines in simulation.

desk verdict Genuinely useful scheduler for multi-loop NCS, but the abstract's infinite-horizon optimality claim is not supported by the truncated-MDP proof. read the letter →

arxiv 1908.01503 v3 pith:TD7GTX6N submitted 2019-08-05 cs.IT cs.NImath.IT

Onur Ayan , Mikhail Vilgelm , Wolfgang Kellerer This is my paper · ORCID

classification cs.ITcs.NImath.IT

keywords ageofinformationnetworkedcontrolsystemsMarkovdecisionprocesswirelessschedulingdiscountedinfinitehorizonvalueiterationestimationerrorpenalty

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a centralized wireless scheduler for multiple feedback control loops can be designed directly around the control cost, using age-of-information (AoI) as the only state information. It models the scheduling problem as a Markov decision process in which each loop's mean-square estimation error is written as a function of its AoI, then applies value iteration to obtain a stationary, deterministic, discounted-optimal policy. The claimed payoff is lower average quadratic estimation error than the state-of-the-art age-optimal scheduler and greedy error scheduler, without any communication beyond each loop's current age. A sympathetic reader would care because this is a concrete step toward making AoI a true cross-layer interface for control, not just a freshness metric.

What carries the argument

The carrying object is the coupled pair of an augmented MDP and a control-aware age penalty. The state is the vector of per-loop ages $\Delta_i[t]$, capped at $M$, and the cost is the mean-square estimation error $g_i(\Delta_i[t])$, which is increasing in age because the open-loop prediction error accumulates through the unstable plant dynamics. Transitions are Bernoulli packet successes, and any probability mass that would leave the finite state space at the cap is folded back onto the cap state. Value iteration on the Bellman equation then produces the stationary policy.

What would settle it

Run the same five-loop scenario with $M$ doubled, for example $M=50$, and compare the value functions and simulated average error; if the policy changes substantially or the error rises, the truncation has not converged. More directly, for a small case with $N=1$ or $N=2$, enumerate policies on the untruncated state space, compute their exact discounted costs, and check whether the truncated policy's Bellman optimality equation is violated by more than the convergence threshold at any reachable state.

Watch

Extended reading notes

Core claim

For a given discount factor $\gamma$ and augmentation level $M$, the paper claims that Algorithm 1 returns a stationary deterministic policy that minimizes the discounted infinite-horizon sum of network-induced mean-square estimation errors for the finite-state augmented MDP, with cost $g_i(\Delta_i[t]) = \sum_{r=0}^{\Delta_i[t]-1} \operatorname{tr}\big((A_i^T)^r A_i^r \Sigma_i\big)$ per loop. The paper also claims, based on Monte Carlo simulation, that this discounted error scheduler (DES) attains lower average quadratic error than the $\gamma$-optimal AoI scheduler and the greedy error scheduler in the five-loop, single-resource scenario it studies.

Load-bearing premise

The scheduler's optimality relies on the assumption that truncating very large ages at a bound $M$ and folding the lost probability back into that bound does not materially change which decisions are best.

Editorial extensions

If this is right

The scheduler can be computed offline once the plant matrices, noise covariances, and packet success probabilities are known, and then applied online using only each loop's current AoI.
Increasing the discount factor makes the scheduler more farsighted and, in the simulated setup, monotonically lowers the average quadratic error, suggesting that long-horizon planning matters for control quality.
The control-aware age penalty, rather than raw age, is what drives the performance gain: the age-optimal baseline achieves the best freshness but much worse estimation error.
The augmentation level $M$ is a genuine complexity-performance trade-off: larger $M$ improves the approximation, but the improvement saturates, so a moderate $M$ can capture most of the benefit.
The same MDP formulation can absorb additional per-action costs, such as communication energy or resource prices, without changing the solution method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If a formal optimality gap for the truncation could be established, the same value-iteration design would justify itself beyond the specific five-loop scenario; the paper currently relies on simulation to show that the approximation is faithful enough.
A natural testable extension is the undiscounted average-cost regime, where the discount factor is removed; the paper's concave error curves suggest the discounted problem may not directly extrapolate to the average-cost optimum.
The method's state-space size $M^N$ makes the exact value-iteration approach expensive for many loops; a reader might infer that factored or approximate dynamic programming over per-loop age distributions is the next step.
Because the cost $g_i$ is derived purely from open-loop prediction error, the same scheduler design could be applied to any remote estimation task whose error grows monotonically with staleness, not just LTI control loops.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper is worth engaging, but the abstract promises more than the math delivers. The finite-state scheduler is solid; the infinite-horizon claim is not.

What's actually new: formulation of multi-loop NCS scheduling as a discounted MDP with AoI states and control-error costs, solved by value iteration. The cost derivation from estimation error (eq. 6) is standard and clean. The comparison against an AoI-only scheduler and a greedy error scheduler is sensible, and the simulations are thorough enough to reproduce. I also appreciate the honest footnote admitting that the algorithm solves the approximated problem, and the discussion of M in Sec IV-D showing the truncation level changes results.

The soft spot is exactly that gap. Algorithm 1 computes the optimal policy for the truncated MDP_M. To license the abstract's claim of minimizing control error over an infinite horizon, you need the ATAS value functions to converge to the original value function or an explicit suboptimality bound. The paper cites Sennott but doesn't verify the conditions for this specific transition law and cost. That's not a nitpick: g_i(Δ) grows like A_i^{2Δ}, so for unstable A_i and γ close to 1 the infinite discounted sum may not even be finite. The fact that M=15 vs M=25 changes the policy (Fig. 7) confirms the truncation matters. Without a convergence result, the unqualified abstract and the 'first work' claim overshoot.

This is fixable. The cleanest response is to soften the claims: state the result as optimal for the augmented finite MDP, and present the infinite-horizon result as a heuristic with simulation support. Alternatively, add a sufficient condition or a numerical gap. Either way, the core contribution—using control-aware age costs in an MDP scheduler—remains useful.

Who this is for: people working on AoI-based scheduling for networked control, and anyone who wants a benchmark of control-aware scheduling. It deserves a serious referee, because the problem is well-motivated and the method is mostly sound. I'd send it to review with the expectation that the authors will need to either add a bound or rewrite the claims.

Referee Report

3 major / 6 minor

Summary. The paper considers a centralized wireless scheduler that, at each slot, selects up to R of N sensors to transmit state measurements to their controllers over an erasure channel. It formulates the problem as a discounted infinite-horizon MDP whose state is the vector of ages of information (AoI) and whose per-step cost is the sum of per-loop mean-square estimation errors g_i(Δ_i), derived for LTI plants with Gaussian disturbances. Since the state space is countably infinite, the authors introduce a finite 'augmentation type approximating sequence' by truncating each AoI at M, and apply value iteration to obtain a stationary deterministic policy (DES). They claim this policy minimizes the infinite-horizon control error and simulate it against a γ-optimal AoI scheduler and a greedy error scheduler. The finite-state value iteration itself is standard, but the paper does not prove that the truncated problem approximates the original infinite-horizon problem.

Significance. If the optimality claim could be substantiated, the paper would be a useful contribution: it gives an explicit control-aware age penalty, a clean MDP formulation, and a systematic DP approach to multi-loop scheduling. The simulation study is also reasonably transparent, including a sensitivity analysis over M, and footnote 3 is honest about the approximation. However, the main advertised result, infinite-horizon optimality, is not established, and the benchmark scheduling rule is mis-specified in Eq. (13). The contribution is therefore currently a heuristic finite-state scheduler with promising simulations, not a proven optimal scheduler for the original problem. The value of the paper would increase substantially if the authors added a formal ATAS convergence argument or an explicit suboptimality bound, and corrected the benchmark definition.

major comments (3)

[§III-A, §III-B, Abstract] The central claim that Algorithm 1 minimizes the infinite-horizon cost Jπ(s0) in (8) is not proven. Algorithm 1 solves the finite MDP_M of Section III-A, and footnote 3 concedes that it solves the approximated problem optimally for a given γ and an augmentation level M. The paper cites Sennott [16] for the ATAS framework but never verifies that the specific transition law (2) and cost (6) satisfy the conditions under which the ATAS value functions converge to V*, nor does it give an optimality gap. This is not a technicality: for A_i = 1.9, g_i(Δ) grows like 1.9^{2Δ}, so for γ close to 1 the discounted sum in (8) may be infinite under policies that starve a sub-system; the original problem is not even shown to be finite. Section IV-D, especially Fig. 7, demonstrates that the truncation level materially changes the policy. To support the abstract and title, the authors must either prove convergence and finiteness or restrict all optimality claims to the finite approximated MDP.
[Eq. (13)] Equation (13) defines the greedy error scheduler as a[t] = arg max_{a∈A} { Σ_i p_i g_i(Δ_i[t]) }. The objective is independent of a, so the arg max is not well-defined; as written every admissible action is a maximizer. The text says the scheduler prioritizes sub-systems with the highest mean squared error, so the intended expression presumably includes the scheduling variables, e.g., a[t] = arg max_{a∈A} Σ_i a_i p_i g_i(Δ_i[t]). Because Fig. 3 and the simulations depend on this benchmark, the definition must be corrected before the comparison claims in Section IV-C can be evaluated.
[Algorithm 1, line 10] In Algorithm 1, the extracted policy is π*(s) ← arg min_{a∈A} J(s). Since J(s) does not depend on a, this does not define a policy; the greedy action should be computed from the Bellman expression in (11), e.g., π*(s) ← arg min_{a∈A} { C(s,a) + γ Σ_{s'} P_{ss'}(a) J(s') }. The sentence before the algorithm states this correctly, but the algorithm as printed is not implementable and must be fixed.

minor comments (6)

[§II-D] The statement 'ai[t] = qi[t], ∀i ∈ N+' uses N+ as an index set; it should be '∀i = 1, ..., N' to avoid confusion with the state-space notation.
[§II-C, Eq. (4)] The conditional expectation in (4) is missing a closing bracket: it should read E[xi[t] | xi[t−Δ_i[t]]].
[§III-A, Definition 2] The redistribution distribution f_{s'}(s,a,ŝ,M) is not specified; the clamping rule used in Fig. 2 should be stated formally as an instance of (9).
[§IV-C] The phrase 'DES results in a concave shape with respect to Δ as γ increases' should read 'with respect to γ' or 'as a function of γ'.
[§III-B] The statement that value iteration converges for any initial value function J0 is only valid for the finite-state discounted problem with bounded costs; this should be stated explicitly, since the original infinite-state problem may have unbounded costs.
[§IV-B] The stopping threshold θ = 0.1 is quite large relative to the costs (of order 10^2) and no discussion of the induced value-function error is given; a smaller threshold or an error bound would make the reported policies more reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the finite-MDP optimal policy is produced by standard value iteration, and the infinite-horizon optimality gap is an unverified approximation claim, not a circular reduction.

full rationale

The derivation is self-contained. Equations (4)-(6) compute the mean-square estimation error from the LTI plant model (3); although the text says "We leverage our results in [12]", the identity E[||e_i||^2] = sum tr((A_i^T)^r A_i^r Sigma_i) is derived in the paper and does not require [12] to be valid. The finite MDP_M is constructed from the original transition (2) by clamping ages and redistributing excess probability (Def. 2), and Algorithm 1 applies standard value iteration whose convergence to the optimal stationary policy of MDP_M follows from Bertsekas [15]. That optimality is therefore a genuine result, not an input. The paper explicitly limits itself in footnote 3 to "the approximated problem optimally for a given gamma and an augmentation level M", and Section IV-D shows M affects outcomes; the lack of an ATAS convergence proof or optimality gap for the original countable-state problem is a formal gap in the infinite-horizon claim, but it is not a case where the conclusion is equivalent to the premise by construction. Finally, the simulation metric (14) is the same g_i(Delta) used in the MDP cost, but this is the objective being optimized rather than a fitted parameter renamed as a prediction; comparing DES, AoIS, and GES on this metric is a legitimate empirical evaluation. No circular step is present.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard MDP machinery plus a handful of modeling assumptions. The main free parameter is the truncation level M. The most fragile axiom is the ATAS approximation quality, which is assumed without formal verification. The error-age relation (6) is taken from the authors' own prior work rather than derived here. No new physical or algorithmic entity is postulated beyond the augmented age state, which is a modeling device, not an invented entity in the sense of a new particle or force.

free parameters (1)

Augmentation level M = 25
State-space truncation level for the finite-state MDP. Chosen from {15,16,17,20,25} based on diminishing performance gains in Fig. 7; it is hand-chosen, not derived from data, and controls the fidelity of the approximation.

assumptions (5)

domain assumption Each plant is stabilizable and a stabilizing feedback gain Li exists.
Footnote 2, Section II-C; required for the control law ui[t] = -Li xhat_i[t] to be valid.
domain assumption The estimator knows the full history of control inputs applied to the plant.
Footnote 1, Section II-C; required for the state prediction (4) and the error evolution (5).
domain assumption The mean-square estimation error as a function of AoI is given by gi(Delta) in (6), adopted from the authors' prior work [12].
Equation (6) is not re-derived; it is imported from [12] and is both the MDP cost and the evaluation metric.
domain assumption The finite-state truncation (ATAS) with excess probability redistribution preserves the essential structure of the original infinite-state MDP for the chosen M.
Section III-A uses Sennott's ATAS without verifying convergence conditions or providing an error bound; Section IV-D gives only empirical M-sensitivity evidence.
domain assumption Transmissions are independent Bernoulli with constant success probabilities pi and constant one-slot delay.
Network model in Section II-A; a standard simplifying model for the analysis.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Optimal Scheduling for Discounted Age Penalty Minimization in Multi-Loop Networked Control." pith.science (2026). https://pith.science/paper/TD7GTX6N

@misc{pith2026190801503,
  author       = {Pith},
  title        = {Pith review of: Optimal Scheduling for Discounted Age Penalty Minimization in Multi-Loop Networked Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TD7GTX6N}},
  note         = {Machine review of arXiv:1908.01503}
}

read the original abstract

Age-of-information (AoI) is a metric quantifying information freshness at the receiver. Since AoI combines packet generation frequency, packet loss, and delay into a single metric, it has received a lot of research attention as an interface between communication network and application. In this work, we apply AoI to the problem of wireless scheduling for multi-loop networked control systems (NCS), i.e., feedback control loops closed over a shared wireless network. We model the scheduling problem as a Markov decision process (MDP) with AoI as its observable states and derive a relation of control system error and AoI. We further derive a stationary scheduling policy to minimize control error over an infinite horizon. We show that our scheduler outperforms the state-of-the-art scheduling policies for NCS. To the best of our knowledge, this is the first work proposing an AoI-based wireless scheduling policy that minimizes the control error over an infinite horizon for multi-loop NCS.

Figures

Figures reproduced from arXiv: 1908.01503 by the authors.

**Figure 1.** Considered scenario with N heterogeneous LTI networked control systems. Solid lines indicate ideal controller-to-plant and plant-to-sensor links. Sensor-to-controller link is closed over a shared wireless channel. Medium access is granted by a scheduler. The smallest time unit we consider is a time slot which is also equal to the sampling period of sub-systems. In other words, when a sensor is granted channel access… view at source ↗

**Figure 2.** Augmentation type approximating sequence for the exem [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Scheduling policies of (a) AoIS, (b) GES, and (c) DES [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Achieved average age-of-information, i.e., [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Illustration of network resources shares for different sub [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 7.** Figure 7: Achieved average quadratic error vs. discount factor [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: Illustration of network resources shares for different sub [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 15 canonical work pages

[16]

L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems. John Wiley & Sons, Inc., 1999

work page 1999
[1]

Cyber physical systems: Design challenges,

E. A. Lee, “Cyber physical systems: Design challenges,” in 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC) , 2008, pp. 363–369

work page 2008
[2]

Real-time status: How often should one update?

S. Kaul, R. Yates, and M. Gruteser, “Real-time status: How often should one update?” in Proceedings IEEE INFOCOM , 2012, pp. 2731–2735

work page 2012
[3]

Status updates through queues,

S. K. Kaul, R. D. Yates, and M. Gruteser, “Status updates through queues,” in 46th Annual Conference on Information Sciences and Systems (CISS), 2012, pp. 1–6

work page 2012
[4]

Age of information with packet management,

M. Costa, M. Codreanu, and A. Ephremides, “Age of information with packet management,” in 2014 IEEE International Symposium on Information Theory, 2014, pp. 1583–1587

work page 2014
[5]

Optimizing age-of-information in a multi- class queueing system,

L. Huang and E. Modiano, “Optimizing age-of-information in a multi- class queueing system,” in IEEE International Symposium on Informa- tion Theory (ISIT) , 2015, pp. 1681–1685

work page 2015
[6]

Update or wait: How to keep your data fresh,

Y . Sun, E. Uysal-Biyikoglu, R. D. Yates, C. E. Koksal, and N. B. Shroff, “Update or wait: How to keep your data fresh,” IEEE Transactions on Information Theory, vol. 63, no. 11, pp. 7492–7508, 2017

2017
[7]

Age-optimal information updates in multihop networks,

A. M. Bedewy, Y . Sun, and N. B. Shroff, “Age-optimal information updates in multihop networks,” in IEEE International Symposium on Information Theory (ISIT) , 2017, pp. 576–580

work page 2017

Show all 16 references

[8]

Minimizing age-of-information in multi-hop wireless networks,

R. Talak, S. Karaman, and E. Modiano, “Minimizing age-of-information in multi-hop wireless networks,” in 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2017, pp. 486–493

2017
[9]

Minimizing the age of information in broadcast wireless networks,

I. Kadota, E. Uysal-Biyikoglu, R. Singh, and E. Modiano, “Minimizing the age of information in broadcast wireless networks,” in 54th An- nual Allerton Conference on Communication, Control, and Computing (Allerton), 2016, pp. 844–851

2016
[10]

Age of information: Design and analysis of optimal scheduling algorithms,

Y . Hsu, E. Modiano, and L. Duan, “Age of information: Design and analysis of optimal scheduling algorithms,” in 2017 IEEE International Symposium on Information Theory (ISIT) , 2017, pp. 561–565

2017
[11]

Age and value of information: Non-linear age case,

A. Kosta, N. Pappas, A. Ephremides, and V . Angelakis, “Age and value of information: Non-linear age case,” in IEEE International Symposium on Information Theory (ISIT) , 2017, pp. 326–330

2017
[12]

Age-of- information vs. value-of-information scheduling for cellular networked control systems,

O. Ayan, M. Vilgelm, M. Kl ¨ugel, S. Hirche, and W. Kellerer, “Age-of- information vs. value-of-information scheduling for cellular networked control systems,” in Proceedings of the 10th ACM/IEEE International Conference on Cyber-Physical Systems , 2019, pp. 109–117

2019
[13]

Aoi-penalty minimization for networked control systems with packet loss,

M. Kl ¨ugel, M. H. Mamduhi, S. Hirche, and W. Kellerer, “Aoi-penalty minimization for networked control systems with packet loss,” 2019

2019
[14]

Control-aware uplink resource allocation for cyber-physical systems in wireless networks,

M. Vilgelm, O. Ayan, S. Zoppi, and W. Kellerer, “Control-aware uplink resource allocation for cyber-physical systems in wireless networks,” in 23th European Wireless Conference , 2017, pp. 1–7

2017
[15]

D. P. Bertsekas, Dynamic Programming and Optimal Control Vol. I and II. Athena Scientiﬁc, 2017

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [16]

L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems. John Wiley & Sons, Inc., 1999

work page 1999

[2] [1]

Cyber physical systems: Design challenges,

E. A. Lee, “Cyber physical systems: Design challenges,” in 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC) , 2008, pp. 363–369

work page 2008

[3] [2]

Real-time status: How often should one update?

S. Kaul, R. Yates, and M. Gruteser, “Real-time status: How often should one update?” in Proceedings IEEE INFOCOM , 2012, pp. 2731–2735

work page 2012

[4] [3]

Status updates through queues,

S. K. Kaul, R. D. Yates, and M. Gruteser, “Status updates through queues,” in 46th Annual Conference on Information Sciences and Systems (CISS), 2012, pp. 1–6

work page 2012

[5] [4]

Age of information with packet management,

M. Costa, M. Codreanu, and A. Ephremides, “Age of information with packet management,” in 2014 IEEE International Symposium on Information Theory, 2014, pp. 1583–1587

work page 2014

[6] [5]

Optimizing age-of-information in a multi- class queueing system,

L. Huang and E. Modiano, “Optimizing age-of-information in a multi- class queueing system,” in IEEE International Symposium on Informa- tion Theory (ISIT) , 2015, pp. 1681–1685

work page 2015

[7] [6]

Update or wait: How to keep your data fresh,

Y . Sun, E. Uysal-Biyikoglu, R. D. Yates, C. E. Koksal, and N. B. Shroff, “Update or wait: How to keep your data fresh,” IEEE Transactions on Information Theory, vol. 63, no. 11, pp. 7492–7508, 2017

2017

[8] [7]

Age-optimal information updates in multihop networks,

A. M. Bedewy, Y . Sun, and N. B. Shroff, “Age-optimal information updates in multihop networks,” in IEEE International Symposium on Information Theory (ISIT) , 2017, pp. 576–580

work page 2017