Online Planning for Decentralized Stochastic Control with Partial History Sharing

Erik Miehling; Kaiqing Zhang; Tamer Ba\c{s}ar

REVIEW 3 major objections 5 minor 30 references

Online Planning for Decentralized Stochastic Control with Partial History Sharing

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Under partial history sharing, decentralized agents can approach the team-optimal value online by each building an identical Monte-Carlo tree from a shared random seed, with no model and no communication.

desk verdict Useful POMCP-for-Dec-POMDP extension whose advertised convergence guarantee only holds for an idealized exact-belief version, not the particle-based algorithm that actually runs. read the letter →

arxiv 1908.02357 v1 pith:IIM2ACRB submitted 2019-08-06 cs.LG cs.AIcs.MAmath.OC

Kaiqing Zhang , Erik Miehling , Tamer Başar This is my paper · ORCID

classification cs.LGcs.AIcs.MAmath.OC

keywords decentralizedstochasticcontrolpartialhistorysharingMonte-CarlotreesearchDec-POMDPcommoninformationapproachonlineplanningcollaborativeintrusionresponsemodel-free

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper targets a long-standing obstacle in decentralized stochastic control: once agents have asymmetric information, the sufficient statistic becomes a distribution over joint histories, and solving the team's problem exactly is intractable. Drawing on the common-information reformulation, the authors argue that under partial history sharing the team's problem is equivalent to a centralized partially observable Markov decision process whose state is the pair of the underlying system state and each agent's local memory. They propose an online Monte-Carlo tree-search algorithm in which every agent builds an identical copy of the same search tree using a shared random seed, eliminating explicit communication and the need for a closed-form model. The central claim is that, given the true common-information belief, the estimated value function converges in probability to the team-optimal value with bias that shrinks as time spent searching grows. The same machinery, the paper shows, contains several existing Dec-POMDP heuristics as special cases, and a collaborative intrusion-response experiment illustrates the algorithm at work.

What carries the argument

The central object is the virtual-history search tree, written $\mathcal{T}(h) = (N(h), V(h))$, whose nodes store the number of visits $N(h)$ and the mean simulated value $V(h)$ for each virtual history $h_t = \{\gamma_1, z_2, \gamma_2, \dots, \gamma_{t-1}, z_t\}$. Branches alternate joint prescriptions $\gamma_t \in \Gamma_t$ and innovations $z_{t+1} \in Z$; a node's children are created when a simulation first reaches that history. Search iterates between selecting prescriptions by the UCB1 rule, $V(h\delta) + \rho\sqrt{\log N(h)/N(h\delta)}$, and expanding the tree with samples from the generative model. The belief is maintained as a set of $K$ particles and updated by rejection sampling: only particles whose sampled innovation matches the true innovation survive. This machinery carries the argument because it converts an intractable multi-agent belief over joint histories into a centralized POMDP belief over a much smaller space, and because the shared random seed makes every agent's copy of the tree exactly the same.

What would settle it

A small PHS instance with a computable exact belief $\pi_\tau$ would settle the issue: if the $K$-particle algorithm's value estimates fail to approach $V^*_\tau(h_\tau)$ as visits grow while $K$ stays fixed, or if rejection sampling fails to collect $K$ particles for a reachable innovation, then the implemented algorithm does not inherit the claimed convergence guarantee.

Watch

Extended reading notes

Core claim

Under partial history sharing, the coordinator's problem—choosing prescriptions that map each agent's local memory to an action—has a belief state $\pi_t = P(x_t, m_t^1, \ldots, m_t^n \mid h_t)$ over the underlying state and all local memories, conditioned on the virtual history $h_t$ of joint prescriptions and innovations. The paper's discovery is that this belief can be used as the root of a POMCP-style (partially observable Monte-Carlo planning) search tree whose nodes are virtual histories and whose branches are joint prescriptions followed by innovations. In the version that assumes the true belief $\pi_\tau$, Lemma 1 (invoking the single-agent POMCP convergence theorem) states that the node value $V(h_\tau)$ converges in probability to $V^*_\tau(h_\tau)$, so the prescription $\hat{\gamma}_t^* = \arg\max_{\delta} V(h_t \delta)$ approximates the team-optimal joint prescription. Decentralization is obtained by a shared random seed: because all agents sample the same particles and generate the same rollouts, their trees are identical and each agent can independently read off its part of the joint prescription. The paper also claims a unifying status for the algorithm: when common information is empty, it reduces to solvers based on non-observable MDPs and occupancy-state MDPs.

Load-bearing premise

The load-bearing premise is that a fixed $K$-particle rejection-sampling belief behaves like the true common-information belief $\pi_\tau$; the convergence proof is stated only for the true belief, and the paper proves no convergence, rate, or error bound for the implemented finite-particle approximation.

Editorial extensions

If this is right

Agents that share partial history can coordinate on a team-optimal joint policy without sending messages, provided they agree on a random seed and have access to a generative simulator.
Because the tree only queries a black-box simulator, the algorithm applies when transition and observation probabilities are unknown but can be sampled, moving beyond model-based planning.
The common-information reformulation shrinks the multi-agent belief to a distribution over the system state and local memories, so instances with substantial shared history may become tractable for online search.
Existing Dec-POMDP heuristics—MAA* tree search and occupancy-state MDP solvers—appear as special cases, giving a single framework for several previously separate algorithms.
In the delayed-sharing intrusion response testbed, higher simulation counts produce lower discounted cost, consistent with the claimed convergence behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the finite-$K$ particle belief is treated as part of the algorithm rather than an implementation detail, a natural next step is to prove whether $K$-particle rejection sampling converges to $\pi_\tau$ and at what rate; a testable intermediate check is to compare the particle belief with exact belief updates on small instances.
The common-seed device assumes agents can agree on and synchronize random number streams; in settings where that is impossible, one would need to replace it with consensus-based tree merging or shared randomness broadcast, and the convergence argument would have to be revisited.
The branching factor of the tree is the number of joint prescriptions, $\prod_i |\mathcal{U}_i|^{|\mathcal{M}_t^i|}$, so scaling to larger multi-agent systems will likely require structure in the prescription space, such as factored action selection or bandit-style sampling over prescriptions.
The collaborative intrusion-response model opens a decentralized treatment of network defense; extending the two-agent experiment to larger attack graphs and more defenders would show whether the empirical convergence persists as the prescription space grows.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the algorithm is a sensible adaptation of POMCP to decentralized POMDPs with partial history sharing, but the paper's headline claim—'provably convergent to team-optimal'—is only proven for an idealized version with the exact belief, not for the K-particle belief the code and experiments actually use.

The genuinely new bit is the combination: using the common-information POMDP reformulation of Nayyar et al. as the planning state, running POMCP-style tree search over joint prescriptions, and using a shared random seed so all agents build identical trees without communication. That's a clean idea, and the paper correctly notes that several existing Dec-POMDP heuristics fall out as special cases. The collaborative intrusion response model is also a reasonable application, and the self-citation to [30] is legitimate—that's the network model they build on.

The soft spots are real and load-bearing. Lemma 1 is Theorem 2 from Silver and Veness, stated for the true belief πτ. But Algorithm 1 maintains B(h), a fixed set of K particles updated by rejection sampling, and the paper gives no argument that the finite-particle belief converges to πτ or that POMCP's value bound survives the approximation. The search tree is driven by samples from B(h), so the convergence theorem simply doesn't apply to the executed algorithm. That's not a minor gap; it's the difference between the abstract's promise and what is actually established. The argmax step is also asserted: value convergence of V(hδ) to V*(hδ) doesn't automatically give convergence of argmax δ V(hδ) to the optimal prescription without extra conditions. Both gaps look fixable—a consistency proof for the particle belief as K→∞, jointly with N→∞, plus a uniform-convergence or separate argument for the argmax—but neither is in the paper.

The experiments don't help: a single small 2-agent instance, no baselines, no comparison to an exact solver, just a cost curve decreasing with simulation count. That's indicative, not demonstrative.

Who should read it: people working on decentralized planning who want a concrete starting point for a sampling-based approach under common information. It deserves a serious referee—the core idea is worth airing—but the referee should insist on a corrected convergence statement, either a proof for the particle-belief version or an honest recasting as a heuristic.

Referee Report

3 major / 5 minor

Summary. The paper addresses decentralized stochastic control with partial history sharing (PHS). It uses the common-information POMDP reduction of Nayyar, Mahajan, and Teneketzis to transform the decentralized problem into a centralized POMDP, then proposes a decentralized online Monte-Carlo tree search algorithm (Algorithm 1) in which each agent builds an identical search tree using a shared random seed and a black-box generative model. The algorithm is claimed to be provably convergent to the team-optimal value function and to require no explicit communication among agents. The paper also relates the algorithm to existing Dec-POMDP solvers (MAA* and the occupancy-state MDP approach) and presents a novel collaborative intrusion response model with a two-agent numerical study showing that the estimated discounted cost decreases as the number of simulations grows.

Significance. If the convergence guarantee were fully established for the implemented algorithm, the paper would make a useful contribution: it combines a known reduction with POMCP-type online search, offers a decentralized execution scheme under a common randomness assumption, and provides a unified perspective on existing Dec-POMDP solvers. The collaborative intrusion response application is a relevant testbed, and the numerical experiment provides a sanity check of the convergence trend, although it does not compare against an optimal baseline. The paper is generally clearly written and the algorithmic construction is sensible. However, the advertised convergence result is only proven for an idealized version of the algorithm that uses exact beliefs, while the executed algorithm uses a finite-particle approximation; the gap is load-bearing for the central claim, and the transfer from value convergence to argmax convergence is also asserted without proof.

major comments (3)

[Section III-C, Lemma 1] Lemma 1 assumes the true belief state πτ is available, but Algorithm 1 (Section III-B) represents the belief by a fixed set of K particles B(h) and updates it by rejection sampling. The paper gives no consistency theorem for the K-particle belief as K grows, no rate of convergence, and no argument that the POMCP-style value bound is preserved under this finite-particle approximation. Because SEARCH draws samples from B(h) rather than from πτ, the stated convergence does not apply to the algorithm whose results are reported in Section IV-B, where K = 400. This is the central gap between the proof and the abstract's claim of a 'provably convergent' algorithm.
[Section III-C, paragraph after Lemma 1] The optimal prescription is chosen as argmaxδ∈Γ V(h_tδ), but the paper does not prove that convergence of the estimated value function V to the true value function V* transfers to convergence of the argmax to the team-optimal joint prescription. Pointwise convergence of values does not by itself imply convergence of the maximizing action unless additional conditions are imposed, such as a unique maximizer with a strictly positive value gap or uniform convergence of the value estimates. This step is asserted rather than derived, and it is essential for the claim that the algorithm yields the approximately optimal control action γ̂^i,*_t(m^i_t).
[Section III-C, Lemma 1] Lemma 1 states convergence 'for any history hτ that are prefixed by ht with τ ≥ t,' which suggests a per-history guarantee at internal nodes of the search tree. The cited theorem from [15] is a root-level convergence result for POMCP under an exact belief state, and the manuscript does not explain how that root-level result extends to arbitrary non-root histories when those histories are reached through particle-based belief updates. Either the lemma must be restricted to the root with the true belief, or the authors need to prove the stronger statement they invoke.

minor comments (5)

[Section III-B] Typo: 'The generate model avoids the need' should read 'The generative model avoids the need.'
[Section IV-A] The detection-probability notation lists δj3 = 0.8 for j = 8, 9, 10, but there are only two alerts (a1 and a2) in the example; this appears to be a typo and should likely read δj2.
[Section III-C] In the statement of Lemma 1, the phrase 'i.e., V(hτ) →p V*τ(hτ)' is awkward; consider writing 'that is, V(hτ) converges in probability to V*τ(hτ).'
[Algorithm 1] In SIMULATE, the child initialization loop uses 'for all γ ∈ Γ' while the subsequent UCB selection uses δ for prescriptions; the notation is inconsistent and could confuse readers.
[Figure 2] The caption does not explain what the gray line represents or how the individual points should be interpreted; please state that the gray line is the mean over sample paths and clarify the error bars or spread.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the central convergence claim is imported from independent external results, and the paper's self-citations are not load-bearing.

full rationale

The paper's derivation chain is: (i) the decentralized PHS problem is reformulated as a centralized POMDP using the structural results of Nayyar, Mahajan, and Teneketzis [5]; (ii) the centralized POMDP is solved online by adapting POMCP [15]; and (iii) convergence is asserted by invoking Theorem 2 of [15] in Lemma 1. None of these load-bearing steps is equivalent to the paper's own inputs by construction. [5] and [15] are independent, externally published results; neither is authored by the present paper's authors, and neither contains the present paper's target claim as a definitional consequence. The paper's self-citations, e.g., [14], [22], [23], and [30], concern related multi-agent learning work and the cyber-network application model; they are not used to justify the convergence theorem or to define away any quantity being predicted. The numerical study varies the simulation count and reports cost, with no fitted parameter being renamed as a prediction. The one substantive weakness, noted in the skeptic's take, is that the implemented algorithm uses a finite K-particle belief approximation while Lemma 1 assumes the true belief state; this is a possible correctness or proof-completeness gap, but it is not a circularity. The advertised convergence is conditional on an idealized version of the algorithm, but the paper does not hide that condition: Lemma 1 explicitly says 'Given the true belief state πτ'. No equation or definition in the paper reduces a derived quantity to an input in a self-referential way, and no load-bearing argument rests on a self-citation. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper contributes the algorithm and application, but the central guarantee rests on imported structural and convergence theorems. No parameters are fitted to data; the exploration constant, particle count, and discount threshold are hand-set for the experiment. The decentralized equivalence depends on a correlation device (shared random seed), which is an assumption introduced by the paper.

free parameters (3)

Exploration constant ρ = 10 (in experiments)
UCB exploration weight in Algorithm 1; hand-chosen for the numerical study, not fitted to data.
Particle count K = 400
Number of particles representing the belief in Section III-B; hand-set, no sensitivity analysis.
Discount horizon threshold ε = 0.1
Stops Monte-Carlo rollouts when β^d < ε; hand-set for the experiment.

assumptions (5)

domain assumption The coordinator's PHS problem is a centralized POMDP with state (x, m^1, ..., m^n), actions being joint prescriptions, and belief π_t = P(x, m | h_t) (Lemma 1 of [5]).
Imported from Nayyar et al. 2013; the whole algorithmic construction operates inside this POMDP and does not reprove the reduction.
standard math The single-agent POMCP value function converges in probability to the optimal value function given the true belief state (Theorem 2 of [15]).
Imported as Lemma 1; the paper transfers this result to the decentralized setting without verifying all hypotheses (e.g., true belief vs particle approximation).
domain assumption Each agent has access to a generative model G that samples exactly from the true dynamics and observation distribution.
Required by Algorithm 1; if G is approximate, the convergence argument collapses.
ad hoc to paper All agents share a common source of randomness and identical copies of G and the belief representation, so their search trees are identical.
Introduced in Section III-B to avoid explicit communication; it is a correlation device. Without identical sampling, agents would compute different prescriptions.
domain assumption State, observation, action, and memory spaces are finite, so the prescription space Γ_t is finite.
Required for the combinatorial tree over prescriptions to be finite; stated in Section II-B.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Online Planning for Decentralized Stochastic Control with Partial History Sharing." pith.science (2026). https://pith.science/paper/IIM2ACRB

@misc{pith2026190802357,
  author       = {Pith},
  title        = {Pith review of: Online Planning for Decentralized Stochastic Control with Partial History Sharing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IIM2ACRB}},
  note         = {Machine review of arXiv:1908.02357}
}

read the original abstract

In decentralized stochastic control, standard approaches for sequential decision-making, e.g. dynamic programming, quickly become intractable due to the need to maintain a complex information state. Computational challenges are further compounded if agents do not possess complete model knowledge. In this paper, we take advantage of the fact that in many problems agents share some common information, or history, termed partial history sharing. Under this information structure the policy search space is greatly reduced. We propose a provably convergent, online tree-search based algorithm that does not require a closed-form model or explicit communication among agents. Interestingly, our algorithm can be viewed as a generalization of several existing heuristic solvers for decentralized partially observable Markov decision processes. To demonstrate the applicability of the model, we propose a novel collaborative intrusion response model, where multiple agents (defenders) possessing asymmetric information aim to collaboratively defend a computer network. Numerical results demonstrate the performance of our algorithm.

Figures

Figures reproduced from arXiv: 1908.02357 by the authors.

**Figure 1.** The dependency graph of the example. The graph consists of ten exploits (e.g. e1 = (∅, {s1}), e4 = ({s1, s2}, {s5}), etc.) and nine security conditions, with goal conditions s8 and s9 represented by double-encircled nodes. j = 8, 9, 10, zero otherwise. Probabilities of false alarm are ζ1 = ζ2 = 0.3. Each agent’s observation space is defined as Y i = {0, 1} designating the presence of an alert from its IDS. The insta… view at source ↗

**Figure 2.** Discounted cost for a given time-step (t = 5) under various simulation counts. Each point represents the cost at t for a distinct sample path for the given value of nsim. The gray line represents the mean discounted cost. B. Numerical Results We investigate the quality of the resulting policy, as computed by Algorithm 1, as a function of the number of simulations used to compute each action, denoted by nsim. We assu… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 28 canonical work pages

[30]

A POMDP approach to the dynamic defense of large-scale cyber networks,

E. Miehling, M. Rasouli, and D. Teneketzis, “A POMDP approach to the dynamic defense of large-scale cyber networks,” IEEE Trans. Inf. Forensics Security, vol. 13, no. 10, pp. 2490–2505, 2018

work page 2018
[15]

Monte-Carlo planning in large POMDPs,

D. Silver and J. Veness, “Monte-Carlo planning in large POMDPs,” in Advances in Neural Inf. Process. Syst. , 2010, pp. 2164–2172

work page 2010
[1]

Optimal control strategies in delayed sharing information structures,

A. Nayyar, A. Mahajan, and D. Teneketzis, “Optimal control strategies in delayed sharing information structures,” IEEE Trans. Automat. Contr., vol. 56, no. 7, pp. 1606–1620, 2011

work page 2011
[2]

Separation of estimation and control for discrete time systems,

H. S. Witsenhausen, “Separation of estimation and control for discrete time systems,”Proc. of the IEEE, vol. 59, no. 11, pp. 1557–1566, 1971

work page 1971
[3]

Sufﬁcient statistics in the optimum control of stochastic systems,

C. Striebel, “Sufﬁcient statistics in the optimum control of stochastic systems,” J. of Math Anal. & Appl., vol. 12, no. 3, pp. 576–592, 1965

work page 1965
[4]

P. R. Kumar and P. Varaiya, Stochastic Systems: Estimation, Identiﬁ- cation, and Adaptive Control . Prentice Hall, NJ, 1986

work page 1986
[5]

Decentralized stochastic control with partial history sharing: A common information approach,

A. Nayyar, A. Mahajan, and D. Teneketzis, “Decentralized stochastic control with partial history sharing: A common information approach,” IEEE Trans. Automat. Contr., vol. 58, no. 7, pp. 1644–1658, 2013

work page 2013
[6]

Dec-POMDPs as non-observable MDPs,

F. A. Oliehoek and C. Amato, “Dec-POMDPs as non-observable MDPs,” IAS Tech. Rep., no. IAS-UV A-14-01, 2014

work page 2014

Show all 30 references

[7]

Dynamic program- ming for partially observable stochastic games,

E. A. Hansen, D. S. Bernstein, and S. Zilberstein, “Dynamic program- ming for partially observable stochastic games,” in Proc. AAAI Conf. on Artiﬁcial Intell. , vol. 4, 2004, pp. 709–715

2004
[8]

Improved memory-bounded dynamic programming for decentralized POMDPs,

S. Seuken and S. Zilberstein, “Improved memory-bounded dynamic programming for decentralized POMDPs,” in Proc. Conf. on Uncer- tainty in Artiﬁcial Intell. , 2007

2007
[9]

Sufﬁcient plan-time statistics for decentralized POMDPs

F. A. Oliehoek, “Sufﬁcient plan-time statistics for decentralized POMDPs.” in Proc. Int. Joint Conf. on Artiﬁcial Intell. , 2013, pp. 302–308

2013
[10]

Optimally solving Dec-POMDPs as continuous-state MDPs,

J. S. Dibangoye, C. Amato, O. Buffet, and F. Charpillet, “Optimally solving Dec-POMDPs as continuous-state MDPs,” J. of Artiﬁcial Intell. Res., vol. 55, pp. 443–497, 2016

2016
[11]

Y ¨uksel and T

S. Y ¨uksel and T. Bas ¸ar, Stochastic Networked Control Systems . Birkh¨auser/Springer, 2013, vol. 10

2013
[12]

Learning to act in decentralized partially observable MDPs,

J. Dibangoye and O. Buffet, “Learning to act in decentralized partially observable MDPs,” in Proc. Int. Conf. on Mach. Learning , 2018, pp. 1233–1242

2018
[13]

Internet of vehicles: From intelligent grid to autonomous cars and vehicular clouds,

M. Gerla, E.-K. Lee, G. Pau, and U. Lee, “Internet of vehicles: From intelligent grid to autonomous cars and vehicular clouds,” in Internet of Things, IEEE World Forum on . IEEE, 2014, pp. 241–246

2014
[14]

Fully decen- tralized multi-agent reinforcement learning with networked agents,

K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Bas ¸ar, “Fully decen- tralized multi-agent reinforcement learning with networked agents,” in Proc. Int. Conf. on Mach. Learning , 2018, pp. 5872–5881

2018
[16]

MAA*: A heuristic search algorithm for solving decentralized POMDPs,

D. Szer, F. Charpillet, and S. Zilberstein, “MAA*: A heuristic search algorithm for solving decentralized POMDPs,” in Proc. Conf. on Uncertainty in Artiﬁcial Intell. , 2005, pp. 576–583

2005
[17]

Monte-carlo expectation maximization for decentralized POMDPs

F. Wu, S. Zilberstein, and N. R. Jennings, “Monte-carlo expectation maximization for decentralized POMDPs.” in Proc. Int. Joint Conf. on Artiﬁcial Intell., 2013, pp. 397–403

2013
[18]

Scalable planning and learning for multiagent POMDPs,

C. Amato, F. A. Oliehoek et al., “Scalable planning and learning for multiagent POMDPs,” in Proc. AAAI Conf. on Artiﬁcial Intell. , 2015, pp. 1995–2002

2015
[19]

Dec-MCTS: Decentralized planning for multi-robot active perception,

G. Best, O. M. Cliff, T. Patten, R. R. Mettu, and R. Fitch, “Dec-MCTS: Decentralized planning for multi-robot active perception,” The Int. J. of Robotics Res. , pp. 1–22, 2018

2018
[20]

Reinforcement learning in decentral- ized stochastic control systems with partial history sharing,

J. Arabneydi and A. Mahajan, “Reinforcement learning in decentral- ized stochastic control systems with partial history sharing,” in Proc. Amer. Contr. Conf. IEEE, 2015, pp. 5449–5456

2015
[21]

Tree-based solution methods for multiagent POMDPs with delayed communication,

F. A. Oliehoek and M. T. Spaan, “Tree-based solution methods for multiagent POMDPs with delayed communication,” in Proc. AAAI Conf. on Artiﬁcial Intell. , 2012

2012
[22]

Networked multi-agent reinforce- ment learning in continuous spaces,

K. Zhang, Z. Yang, and T. Bas ¸ar, “Networked multi-agent reinforce- ment learning in continuous spaces,” in Proc. Conf. on Decision and Contr. IEEE, 2018

2018
[23]

Finite-sample analyses for fully decentralized multi-agent reinforcement learning,

K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Bas ¸ar, “Finite-sample analyses for fully decentralized multi-agent reinforcement learning,” arXiv preprint arXiv:1812.02783 , 2018

2018 arXiv
[24]

Policy iteration for decentralized control of Markov decision processes,

D. S. Bernstein, C. Amato, E. A. Hansen, and S. Zilberstein, “Policy iteration for decentralized control of Markov decision processes,” J. of Artiﬁcial Intell. Res. , vol. 34, pp. 89–132, 2009

2009
[25]

Bas ¸ar and G

T. Bas ¸ar and G. J. Olsder, Dynamic Noncooperative Game Theory . SIAM, 1999, vol. 23

1999
[26]

Decentralized control of partially observable Markov Decision Processes,

C. Amato, G. Chowdhary, A. Geramifard, N. K. Ure, and M. J. Kochenderfer, “Decentralized control of partially observable Markov Decision Processes,” in Proc. Conf. on Decision and Contr. IEEE, 2013, pp. 2398–2405

2013
[27]

Finite-time analysis of the multiarmed bandit problem,

P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,” Mach. Learning , vol. 47, no. 2-3, pp. 235–256, 2002

2002
[28]

Taxonomy and survey of collaborative intrusion detection,

E. Vasilomanolakis, S. Karuppayah, M. M ¨uhlh¨auser, and M. Fischer, “Taxonomy and survey of collaborative intrusion detection,” ACM Comput. Surveys, vol. 47, no. 4, p. 55, 2015

2015
[29]

Scalable, graph-based network vulnerability analysis,

P. Ammann, D. Wijesekera, and S. Kaushik, “Scalable, graph-based network vulnerability analysis,” in Proc. Conf. on Comput. and Com- mun. Security. ACM, 2002, pp. 217–224

2002

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [30]

A POMDP approach to the dynamic defense of large-scale cyber networks,

E. Miehling, M. Rasouli, and D. Teneketzis, “A POMDP approach to the dynamic defense of large-scale cyber networks,” IEEE Trans. Inf. Forensics Security, vol. 13, no. 10, pp. 2490–2505, 2018

work page 2018

[2] [15]

Monte-Carlo planning in large POMDPs,

D. Silver and J. Veness, “Monte-Carlo planning in large POMDPs,” in Advances in Neural Inf. Process. Syst. , 2010, pp. 2164–2172

work page 2010

[3] [1]

Optimal control strategies in delayed sharing information structures,

A. Nayyar, A. Mahajan, and D. Teneketzis, “Optimal control strategies in delayed sharing information structures,” IEEE Trans. Automat. Contr., vol. 56, no. 7, pp. 1606–1620, 2011

work page 2011

[4] [2]

Separation of estimation and control for discrete time systems,

H. S. Witsenhausen, “Separation of estimation and control for discrete time systems,”Proc. of the IEEE, vol. 59, no. 11, pp. 1557–1566, 1971

work page 1971

[5] [3]

Sufﬁcient statistics in the optimum control of stochastic systems,

C. Striebel, “Sufﬁcient statistics in the optimum control of stochastic systems,” J. of Math Anal. & Appl., vol. 12, no. 3, pp. 576–592, 1965

work page 1965

[6] [4]

P. R. Kumar and P. Varaiya, Stochastic Systems: Estimation, Identiﬁ- cation, and Adaptive Control . Prentice Hall, NJ, 1986

work page 1986

[7] [5]

Decentralized stochastic control with partial history sharing: A common information approach,

A. Nayyar, A. Mahajan, and D. Teneketzis, “Decentralized stochastic control with partial history sharing: A common information approach,” IEEE Trans. Automat. Contr., vol. 58, no. 7, pp. 1644–1658, 2013

work page 2013

[8] [6]

Dec-POMDPs as non-observable MDPs,

F. A. Oliehoek and C. Amato, “Dec-POMDPs as non-observable MDPs,” IAS Tech. Rep., no. IAS-UV A-14-01, 2014

work page 2014