Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Feature-Based Belief Aggregation for Partially Observable Markov Decision Problems

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage feature-based aggregation scheme approximates the optimal cost of a POMDP with a uniform error bound $\epsilon/(1-\alpha)$, and with a lower-bound guarantee when representative beliefs span the belief simplex.

desk verdict A clean, practical two-stage aggregation scheme for POMDPs with a correct central error bound; the Prop. 6 sign error is a typo, not a proof killer, but the particle-filter experiments sit outside the theory. read the letter →

arxiv 2507.04646 v1 pith:U5GTAHLM submitted 2025-07-07 eess.SY cs.SY

Yuchao Li , Kim Hammar , Dimitri Bertsekas This is my paper
classification eess.SYcs.SY MSC 90C4093E20
keywords POMDPbeliefaggregationfeature-basedapproximatedynamicprogrammingerrorbounddiscountedcostvalueiterationbiased
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to approximate the optimal cost function of a finite-state, infinite-horizon, discounted POMDP without working directly with high-dimensional belief vectors. The method first groups the unobservable states into a small set of feature states and then aggregates beliefs over those features to a finite set of representative feature beliefs, so the resulting aggregate MDP can be solved by standard dynamic programming. The authors prove that the interpolated approximation $\tilde J$ satisfies $|\tilde J(b)-J^*(b)|\le \epsilon/(1-\alpha)$ at every belief $b$, where $\epsilon$ is the largest variation of the true optimal cost within each footprint set of a representative belief. Under a convex-hull condition on the representative beliefs, they prove the approximation is a lower bound, $0\le J^*(b)-\tilde J(b)\le \epsilon/(1-\alpha)$. They also extend the scheme to biased aggregation, where an existing cost approximation $V$ is folded into the stage costs so the error term becomes the variation of $J^*-V$ instead of $J^*$ itself. If these claims hold, large POMDPs become solvable with rigorous performance certificates rather than relying on heuristics alone.

What carries the argument

The load-bearing object is the aggregate MDP whose state space is the finite grid of representative feature beliefs $\tilde Q$. Two mappings carry the compression: $D$ disaggregates a representative feature belief into a full belief over original states, and $\Phi$ projects any updated belief back onto the feature-belief simplex; composing them with the belief update $F$ gives the one-step transition operator $H$ of Eq. (16), which is monotone and an $\alpha$-contraction in the max norm. Solving $r^*=Hr^*$ by value iteration or asynchronous value iteration yields the cost vector used in the interpolation formula (18). The error analysis works entirely through the footprint sets $S_{\tilde q}$ (beliefs whose interpolation weights attach to $\tilde q$): the bound says the worst error is the largest fluctuation of $J^*$ inside any footprint, amplified by the factor $1/(1-\alpha)$. The lower-bound result leans on concavity of $J^*$ in the belief vector together with the convex-hull condition (42).

What would settle it

Take a small POMDP for which $J^*$ can be computed exactly, such as the paper's treasure-hunting problem with two sites, build a feature aggregation with overlapping footprint sets, and evaluate $|\tilde J(b)-J^*(b)|$ on a dense grid of beliefs. If any evaluation exceeds $\epsilon/(1-\alpha)$ with $\epsilon$ defined as in Eq. (39), Proposition 5 is false; if none does, the bound stands for that instance and its tightness can be measured directly.

Watch

Extended reading notes

Core claim

At the center of the paper is the observation that a POMDP's belief space can be compressed twice without losing a provable handle on optimal cost: first compress the hidden state space into feature states via aggregation and disaggregation probabilities, then compress the feature-belief simplex into a finite grid of representative feature beliefs. Solving the resulting finite MDP gives a cost vector $r^*$, and the interpolation formula $\tilde J(b)=\sum_{\tilde q}\psi_{\Phi(b)\tilde q}r^*_{\tilde q}$ maps those representative costs back to every original belief. The central result, Proposition 5, states that this approximation is uniformly within $\epsilon/(1-\alpha)$ of $J^*$, where $\epsilon$ is the maximum variation of $J^*$ over the footprint sets $S_{\tilde q}=\{b:\psi_{\Phi(b)\tilde q}>0\}$. The bound requires no assumption that footprint sets partition the belief space, which is exactly what lets the feature-based construction overlap and still work. Proposition 6 adds that when the images $\tilde B$ of the representative beliefs span the whole belief simplex through Eq. (42), the approximation is a lower bound for $J^*$, and it is linear when the representative set has $n$ elements. The biased variant replaces $J^*$ by the residual $J^*-V$ in the definition of $\epsilon$, which explains why a good guess $V$ buys accuracy with fewer representative beliefs.

Load-bearing premise

The theoretical error bound assumes the belief update $F$ is computed exactly, but in the two largest experiments it is replaced by a 100-particle rejection-sampling particle filter with no error estimate, so the reported costs are not covered by the bound.

Editorial extensions

If this is right

  • With a fixed representative grid, the approximation error is controlled by how much $J^*$ varies inside footprint sets, so refining the grid where $J^*$ changes most should shrink the bound.
  • Because Proposition 5 does not require footprint sets to be disjoint, features can map many original states to shared feature states, which is what lets the method scale to state spaces with tens of thousands of hidden states.
  • When the convex-hull condition holds, the approximation is a conservative lower bound and can be used safely in rollout or policy-improvement procedures.
  • Biased aggregation reuses any existing cost approximation $V$: the error term becomes the variation of $J^*-V$, so a good $V$ yields comparable accuracy with far fewer representative feature beliefs.
  • The aggregate MDP inherits standard value-iteration convergence, so computational effort scales with the size of the feature grid rather than with the full hidden-state count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The bound suggests an adaptive design loop the paper does not develop: choose feature partitions and representative grids to minimize footprint variation, guided by local estimates of $J^*$ from a smaller exact problem.
  • Because the proof does not rely on POMDP-specific structure, the same two-stage analysis transfers to any large-state MDP with a feature embedding, a point the authors assign to a companion analysis.
  • In the two largest experiments the belief update is replaced by a 100-particle rejection-sampling particle filter, so the reported costs carry particle noise that the $\epsilon/(1-\alpha)$ guarantee does not cover; a direct test would compare exact-filter and particle-filter results on a moderately sized instance.
  • Biased aggregation could be iterated: feed the resulting approximation back in as the next bias function and re-solve, which should drive the residual variation and hence $\epsilon$ toward zero even on a coarse representative grid.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes a two-stage aggregation scheme for infinite-horizon discounted finite-state POMDPs. In the first stage, unobservable states are aggregated into feature states via disaggregation and aggregation distributions; in the second, beliefs over feature states are projected onto a finite set of representative feature beliefs. The resulting aggregate MDP is solved by value iteration, and its fixed point is interpolated to form an approximate cost function J~ used in one-step lookahead. The paper's main theoretical results are Proposition 5, an epsilon/(1-alpha) error bound for |J~ - J*| where epsilon is the maximal variation of J* over footprint sets, and Proposition 6, a lower-bound result under a convex-hull condition (Eq. 42). A biased-aggregation extension (Props. 7-8) modifies stage costs using an approximate value function. Experiments on treasure hunting, Rocksample, and CAGE-2 show competitive costs with lower computational effort.

Significance. If the claims hold, this is a useful and fairly general framework. The error-bound proof is self-contained and does not rely on a partition of the belief space, which relaxes assumptions in the classical Tsitsiklis and Van Roy bound. The lower-bound result gives a clean sufficient condition based on convexity of J* and convex-hull coverage of the belief simplex. The paper ships code and clear algorithmic descriptions, and the experimental comparison is broad. The main caveats are that the bound is not computable a priori (it depends on the unknown J* and footprint geometry), and the large-scale experiments use a particle-filter approximation of the belief update that is outside the assumptions of the theorems; both should be acknowledged and addressed in revision.

major comments (3)
  1. [Section 3.3, Eq. (48)] Eq. (48) in the proof of Proposition 6 has the wrong inequality. The line immediately before it, Eq. (47), states J*(D(q)) >= min_u [ ... ] = (H rhat)(q), i.e., H rhat <= rhat, not H rhat >= rhat. As printed, H rhat >= rhat would imply r* >= rhat under the monotone-contraction argument, which is the opposite of the needed r* <= rhat. Please correct Eq. (48) and the following sentence; with H rhat <= rhat, the monotone-contraction argument indeed gives r* <= rhat, so the lower-bound conclusion is recoverable.
  2. [Appendix A.2 and Section 4] Appendix A.2 states that for RS(10,10) and CAGE-2 the belief estimator F is replaced by a 100-particle rejection-sampling particle filter, while Props. 5-8 assume F is computed exactly. No bias, variance, or consistency estimate for this replacement is provided, so the error bounds of Props. 5-8 do not cover the costs reported in Figs. 10-13. The paper should either restrict the theoretical claims to exact-F settings, add an analysis or sensitivity study for the particle filter, or explicitly designate the large-scale experimental claims as heuristic.
  3. [Section 4 and Appendix A.2] The lower-bound guarantee of Proposition 6 depends critically on the convex-hull condition Eq. (42). The experimental scheme described in Appendix A.2 uses the nearest-neighbor hard-aggregation rule of Eq. (10), which does not satisfy Eq. (42) in the compressed regimes (rho=1 or rho=2) used for RS(7,8), RS(10,10), and CAGE-2. The paper should state that the lower-bound theorem applies to the constructions in Examples 3.3 and 5.1 rather than to these experiments; otherwise readers may attribute a guarantee to the large-scale results that the theory does not provide.
minor comments (5)
  1. [Figure 8] The y-axis label reads '\||J* - J~||'; the stray vertical bar should be removed.
  2. [Section 5.2] Props. 7 and 8 are stated without proofs; since they are central to the biased-aggregation contribution, a brief proof sketch or an explicit reduction to Props. 5-6 via Eq. (50) would improve verifiability.
  3. [Section 3.2] The converse half of Proposition 5 is asserted without display; writing out the symmetric argument would prevent the sign confusion that later appears in Proposition 6.
  4. [Section 2.3] The phrase 'we can see that the unobservable states i involved in our algorithm form a set X-hat' is informal; X-hat is defined after the statement and should be introduced before use.
  5. [Figures 7 and 9] The caption text '10Q, with m=... representative feature beliefs' appears garbled; please restate, for example, as 'the one-dimensional feature-belief simplex Q'.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the aggregation error bound is proven by a self-contained DP contraction argument; self-citations provide framework only.

full rationale

Prop. 5's bound |J~(b) − J*(b)| ≤ ε/(1−α) is not circular: ε is defined as the true optimal cost's variation over the footprint sets (Eq. 39), and the proof establishes Hr ≤ r for the vector r_~q = inf_{b∈S_~q} J*(b) + ε/(1−α), then invokes monotonicity and contractivity of H (Prop. 1, a standard argument) to obtain r* ≤ r and the upper bound, with the symmetric sup-vector giving the lower bound. The bound's content is the contraction factor 1/(1−α) and the honest dependence on J*; Fig. 8 even shows the bound is loose, so it is not manufactured to match the method. Prop. 6's lower-bound claim depends on concavity of J* (cited to Sondik 1978 and Krishnamurthy 2016, both external) and on the explicit convex-hull condition Eq. (42); it does not assume the conclusion J~ ≤ J*. The derivation chain is therefore self-contained and does not reduce to its inputs. Two non-circularity concerns should be noted. First, the displayed inequality H r̂ ≥ r̂ in Eq. (48) has the wrong direction — Eq. (47) yields H r̂ ≤ r̂ — and the claimed implication r̂ ≥ r* follows only with the corrected sign; this is a correctness or typographical issue in the manuscript's own argument, not an equation-level equivalence of input and output. Second, several framework citations are to the authors' own prior work ([Ber12] aggregation, [Ber19a] biased aggregation, [LB25] the companion MDP bound), but the load-bearing identities are re-derived in-paper (e.g., J* = Ṽ + V in Eq. (50)), so the self-citations are not load-bearing reductions. The manuscript itself flags its limitations: Props. 7 and 8 are stated without proofs (Section 5.2), and for RS(10,10) and CAGE-2 the belief estimator F is approximated by a 100-particle rejection-sampling particle filter (Appendix A.2) while Props. 5–8 assume exact F; these are support gaps for the experimental claims, not circularity.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The theorems are derived from standard dynamic programming facts, not from fitted constants. The only hand-chosen quantities are experimental design parameters: resolution rho, feature mappings, and the particle filter size, which influence the reported empirical performance but are not part of the error-bound derivation. No new physical entities are posited.

free parameters (4)
  • discretization resolution rho = rho=10 for treasure hunting; rho=3 for RS(4,4); rho=2 for RS(5,5) and RS(5,7); rho=1 for RS(7,8), RS(10,10) and CAGE-2
    Controls the number of representative feature beliefs in Eq. (9). Chosen per problem by hand; it influences approximation quality and compute time but is not fitted to the theoretical bound.
  • particle filter size = 100
    Used to approximate the belief update F for RS(10,10) and CAGE-2 (Appendix A.2). The approximation error introduced by the particle filter is not quantified.
  • CAGE-2 feature mapping = attacker-state, attacker-target, decoy-state; feature space size 427,500
    Domain-specific design choice described in Appendix A.2. The paper credits this feature choice for the strong CAGE-2 result, but it is not derived from first principles.
  • RS(10,10) feature mapping = 3x3 grid positions; feature space size 9,216
    Hand-designed aggregation of the 10x10 grid positions used to define feature states in Appendix A.2.
assumptions (6)
  • standard math J* is bounded and uniquely satisfies the Bellman equation (4).
    Invoked throughout Section 3 to justify the contraction properties, the fixed point r*, and the use of J* in the error-bound proof.
  • domain assumption The optimal cost function J* is concave over the belief simplex.
    Used in Eq. (45) of Proposition 6 via citations to Sondik, Krishnamurthy, and Bertsekas. This is required for the lower-bound result.
  • domain assumption The feature sets Ix are disjoint and each state is assigned to at most one feature set, with disaggregation and aggregation probabilities satisfying Eqs. (5) and (6).
    Necessary for the injectivity of the disaggregation map D (Prop. 2) and for the inversion identity between D and Phi (Prop. 3).
  • domain assumption The belief aggregation probabilities satisfy Eq. (8), and for the lower-bound theorem the convex hull condition (42) holds.
    Eq. (42) is explicitly assumed in Props. 6 and 8. It is a strong assumption and generally fails when the number of representative beliefs is smaller than the number of unobservable states.
  • domain assumption The bias function V in Section 5 is a bounded approximation to J*.
    Required for the modified stage cost in Eq. (49) and for the finiteness of epsilon in Props. 7 and 8. The paper gives no quantitative guarantee on the accuracy of V.
  • domain assumption The belief update F follows Bayes' rule and can be evaluated exactly in the analytical examples.
    Used in Eq. (24) and in the small-scale experiments. The large-scale experiments replace F with a 100-particle rejection-sampling filter, which is outside the assumptions of the main theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Feature-Based Belief Aggregation for Partially Observable Markov Decision Problems." pith.science (2026). https://pith.science/paper/U5GTAHLM

@misc{pith2026250704646,
  author       = {Pith},
  title        = {Pith review of: Feature-Based Belief Aggregation for Partially Observable Markov Decision Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U5GTAHLM}},
  note         = {Machine review of arXiv:2507.04646}
}
read the original abstract

We consider a finite-state partially observable Markov decision problem (POMDP) with an infinite horizon and a discounted cost, and we propose a new method for computing a cost function approximation that is based on features and aggregation. In particular, using the classical belief-space formulation, we construct a related Markov decision problem (MDP) by first aggregating the unobservable states into feature states, and then introducing representative beliefs over these feature states. This two-stage aggregation approach facilitates the use of dynamic programming methods for solving the aggregate problem and provides additional design flexibility. The optimal cost function of the aggregate problem can in turn be used within an on-line approximation in value space scheme for the original POMDP. We derive a new bound on the approximation error of our scheme. In addition, we establish conditions under which the cost function approximation provides a lower bound for the optimal cost. Finally, we present a biased aggregation approach, which leverages an optimal cost function estimate to improve the quality of the approximation error of the aggregate problem.

Figures

Figures reproduced from arXiv: 2507.04646 by the authors.

Figure 1
Figure 1. The controlled dynamic feature-state system constructed through the [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Our feature-based belief aggregation method: (i) we map the un [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The controlled dynamic belief system constructed through our feature [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Illustration of the interpolation formula given by Eq. (18). It is used [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Approximation error and computation time of the cost function ap [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Schematic illustration of the scalar ϵ of Eq. (39) in Prop. 5; the illustration is based on an approximation with representative feature beliefs q, ˜ q˜ ′ , and ˜q ′′, with footprint sets Sq˜, Sq˜ ′ , and Sq˜ ′′ , respectively; cf. Eq. (35). Note that Prop. 5 places no…
Figure 7
Figure 7. Figure 7: Comparison between the optimal cost function [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Comparison between the theoretical error bound [cf. Prop. 5] and the [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Comparison between the optimal cost function [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Comparison between our method and value-based approximation [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Comparison between our method and heuristic search methods for [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Comparison between our method and policy-based approximation [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Evaluation results on the CAGE-2 POMDP [CAG22]. The upper plot [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: The controlled dynamic belief system constructed through our [PITH_FULL_IMAGE:figures/full_fig_p030_14.png]
Figure 15
Figure 15. Figure 15: Comparison between the optimal cost function [PITH_FULL_IMAGE:figures/full_fig_p031_15.png]
Figure 16
Figure 16. Figure 16: Comparison between the optimal cost function [PITH_FULL_IMAGE:figures/full_fig_p032_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Recovery Control in Replicated Systems through Autonomous Multiagent Rollout

    eess.SY 2026-07 accept novelty 6.0 of 10

    Autonomous multiagent rollout with signaling approximates near-optimal recovery policies for replicated systems, scaling to 70 replicas and lowering cost versus practical baselines.

  2. Online Incident Response Planning under Model Misspecification through Bayesian Learning and Belief Quantization

    cs.LG 2025-08 conditional novelty 5.0 of 10

    MOBAL learns a model of an ongoing cyberattack with Bayesian updates and computes incident responses with a quantized version of that model, giving robustness to model misspecification on CAGE-2.

Reference graph

Works this paper leans on

35 extracted references · 29 canonical work pages · cited by 2 Pith papers

  1. [1]

    Bertsekas

    Dimitri P. Bertsekas. Distributed dynamic programming. IEEE Transactions on Automatic Control , 27(3):610--616, 1982

    work page 1982

  2. [2]

    Bertsekas

    Dimitri P. Bertsekas. Distributed asynchronous computation of fixed points. Mathematical Programming , 27(1):107--120, 1983

    work page 1983

  3. [3]

    Bertsekas

    Dimitri P. Bertsekas. Dynamic Programming and Optimal Control: Vol. II . Athena Scientific Belmont, 4th edition, 2012

    2012

  4. [4]

    Bertsekas

    Dimitri P. Bertsekas. Dynamic Programming and Optimal Control: Vol. I . Athena Scientific Belmont, 4th edition, 2017

    work page 2017

  5. [5]

    Bertsekas

    Dimitri P. Bertsekas. Feature-based aggregation and deep reinforcement learning: A survey and some new implementations. IEEE/CAA Journal of Automatica Sinica , 6(1):1--31, 2018

    2018

  6. [6]

    Bertsekas

    Dimitri P. Bertsekas. Biased aggregation, rollout, and enhanced policy improvement for reinforcement learning. arXiv preprint arXiv:1910.02426 , 2019

    arXiv 1910

  7. [7]

    Bertsekas

    Dimitri P. Bertsekas. Reinforcement Learning and Optimal Control . Athena Scientific, 2019

    2019

  8. [8]

    Bertsekas and John N

    Dimitri P. Bertsekas and John N. Tsitsiklis. Neuro-Dynamic Programming . Athena Scientific Belmont, MA, 1996

    work page 1996

Show all 35 references
  1. [9]

    TTCP CAGE Challenge 2

    CAGE. TTCP CAGE Challenge 2 . In AAAI-22 Workshop on Artificial Intelligence for Cyber Security (AICS) , 2022. https://github.com/cage-challenge/cage-challenge-2

    2022

  2. [10]

    A survey of convergence results on particle filtering methods for practitioners

    Dan Crisan and Arnaud Doucet. A survey of convergence results on particle filtering methods for practitioners . IEEE Transactions on Signal Processing , 50(3):736--746, 2002

    2002

  3. [11]

    Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman

    Karl W. Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In International Conference on Machine Learning , pages 2020--2027. PMLR, 2021

    2020

  4. [12]

    Sunberg, Edward Balaban, Tim A

    Maxim Egorov, Zachary N. Sunberg, Edward Balaban, Tim A. Wheeler, Jayesh K. Gupta, and Mykel J. Kochenderfer. POMDP s.jl: A framework for sequential decision making under uncertainty. Journal of Machine Learning Research , 18(26):1--5, 2017

    2017

  5. [13]

    Lupu, and Dimitri Bertsekas

    Kim Hammar, Yuchao Li, Tansu Alpcan, Emil C. Lupu, and Dimitri Bertsekas. Adaptive network security policies via belief aggregation and rollout, 2025. Working paper, to appear

    2025

  6. [14]

    Software for the paper ``Feature-Based Belief Aggregation for Partially Observable Markov Decision Problems" , 2025

    Kim Hammar, Yuchao Li, and Dimitri Bertsekas. Software for the paper ``Feature-Based Belief Aggregation for Partially Observable Markov Decision Problems" , 2025. Example code is available at https://github.com/Limmen/feature_based_pomdp_aggregation. The CAGE-2 code is availab...

    2025

  7. [15]

    Partially Observed Markov Decision Processes: From Filtering to Controlled Sensing

    Vikram Krishnamurthy. Partially Observed Markov Decision Processes: From Filtering to Controlled Sensing . Cambridge University Press, 2016

    2016

  8. [16]

    An error bound for aggregation in approximate dynamic programming

    Yuchao Li and Dimitri Bertsekas. An error bound for aggregation in approximate dynamic programming. arXiv preprint arXiv:2507.01324 , 2025

    2025 arXiv

  9. [17]

    Littman, Anthony R

    Michael L. Littman, Anthony R. Cassandra, and Leslie Pack Kaelbling. Learning policies for partially observable environments: Scaling up. In Machine Learning Proceedings 1995 , pages 362--370. Elsevier, 1995

    1995

  10. [18]

    William S. Lovejoy. Computationally feasible bounds for partially observed Markov decision processes. Operations Research , 39(1):162--175, 1991

    1991

  11. [19]

    Recurrent model-free RL can be a strong baseline for many POMDP s

    Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many POMDP s. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 16691--16723. P...

    2022

  12. [20]

    Ong, Shao Wei Png, David Hsu, and Wee Sun Lee

    Sylvie C.W. Ong, Shao Wei Png, David Hsu, and Wee Sun Lee. Planning under uncertainty for robotic tasks with mixed observability. The International Journal of Robotics Research , 29(8):1053--1068, 2010

    2010

  13. [21]

    Value-directed compression of POMDPs

    Pascal Poupart and Craig Boutilier. Value-directed compression of POMDPs . In Advances in Neural Information Processing Systems , volume 15. MIT Press, 2002

    2002

  14. [22]

    Anytime point-based approximations for large POMDPs

    Joelle Pineau, Geoffrey Gordon, and Sebastian Thrun. Anytime point-based approximations for large POMDPs . Journal of Artificial Intelligence Research , 27:335--380, 2006

    2006

  15. [23]

    Finding approximate POMDP solutions through belief compression

    Nicholas Roy, Geoffrey Gordon, and Sebastian Thrun. Finding approximate POMDP solutions through belief compression. Journal of artificial intelligence research , 23:1--40, 2005

    2005

  16. [24]

    Sunberg and Mykel J

    Zachary N. Sunberg and Mykel J. Kochenderfer. Online algorithms for POMDP s with continuous state, action, and observation spaces. In Proceedings of the 28th International Conference on Automated Planning and Scheduling (ICAPS) , 2018

    2018

  17. [25]

    Edward J. Sondik. The optimal control of partially observable Markov processes over the infinite horizon: Discounted costs. Operations Research , 26(2):282--304, 1978

    1978

  18. [26]

    Heuristic search value iteration for POMDPs

    Trey Smith and Reid Simmons. Heuristic search value iteration for POMDPs . In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence , UAI '04, page 520–527. AUAI Press, 2004

    2004

  19. [27]

    Monte-Carlo planning in large POMDPs

    David Silver and Joel Veness. Monte-Carlo planning in large POMDPs . In Advances in Neural Information Processing Systems , volume 23, 2010

    2010

  20. [28]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017

    2017 arXiv

  21. [29]

    DESPOT : Online POMDP planning with regularization

    Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. DESPOT : Online POMDP planning with regularization. Advances in neural information processing systems , 26, 2013

    2013

  22. [30]

    On the asymptotic optimality of finite approximations to Markov decision processes with Borel spaces

    Naci Saldi, Serdar Y\" u ksel, and Tam\' a s Linder. On the asymptotic optimality of finite approximations to Markov decision processes with Borel spaces. Mathematics of Operations Research , 42(4):945--978, 2017

    2017

  23. [31]

    Tsitsiklis and Benjamin van Roy

    John N. Tsitsiklis and Benjamin van Roy. Feature-based methods for large scale dynamic programming . Machine Learning , 22(1):59--94, Mar 1996

    1996

  24. [32]

    Adaptive online packing-guided search for POMDPs

    Chenyang Wu, Guoyu Yang, Zongzhang Zhang, Yang Yu, Dong Li, Wulong Liu, and Jianye Hao. Adaptive online packing-guided search for POMDPs . In Advances in Neural Information Processing Systems , volume 34, pages 28419--28430. Curran Associates, Inc., 2021

    2021

  25. [33]

    Bertsekas

    Huizhen Yu and Dimitri P. Bertsekas. Discretized approximations for POMDP with average cost. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence , UAI '04, page 619–627, Arlington, Virginia, USA, 2004. AUAI Press

    2004

  26. [34]

    Rong Zhou and Eric A. Hansen. An improved grid-based approximation algorithm for POMDPs . In International Joint Conference on Artificial Intelligence , 2001

    2001

  27. [35]

    Optimal control of Markov processes with incomplete state information

    Karl Johan Åström. Optimal control of Markov processes with incomplete state information . Journal of Mathematical Analysis and Applications , 10(1):174--205, 1965

    1965

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.