Consistent Variance Estimation for Q-Function Estimators in Finite-Horizon MDP Tree Search

Chun-Hung Chen; Hadi El-Amine; Jie Xu; Michael C. Fu; Zhenyu Yue

REVIEW 4 major objections 4 minor 20 references

Consistent Variance Estimation for Q-Function Estimators in Finite-Horizon MDP Tree Search

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The usual sample-variance formula for Q-function estimates in tree search is biased downward even at infinite sample size; a recursive estimator that propagates successor-state variance is consistent and practical.

desk verdict Real bias result and a clean node-local variance update, but the recursive estimator's consistency claim is load-bearing and unproven; deserves a serious referee, not acceptance as-is. read the letter →

arxiv 2607.14555 v1 pith:VMFIOTOA submitted 2026-07-16 eess.SY cs.SY

Zhenyu Yue , Jie Xu , Chun-Hung Chen , Hadi El-Amine , Michael C. Fu This is my paper · ORCID

classification eess.SYcs.SY MSC 90C4062F1262L20

keywords MonteCarlotreesearchQ-functionestimationvarianceMarkovdecisionprocessfinite-horizonMDPOCBAUCB1-Tunedconsistency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper targets a persistent blind spot in Monte Carlo tree search (MCTS) for finite-horizon Markov decision processes: the uncertainty of a state-action (Q-function) estimate is normally computed as if each simulated path were an independent draw, but paths that reach the same successor state share the same value estimate, so the usual sample variance undercounts the true uncertainty—and the undercount does not vanish as more simulations are collected. The paper proves a variance decomposition, identifies the missing term as the probability-weighted variance of successor value estimates, and builds a recursive estimator that adds that term back. It then gives an implementation that stores only local reward statistics per successor, matching the exact computation without storing full histories. Embedded in two common selection rules (OCBA and UCB1-Tuned), the estimator improves the probability of selecting the optimal first action in inventory-control and kidney-donation test problems.

What carries the argument

The load-bearing object is the recursive variance estimator of Eq. (12): ζ̂²(s,a) = ζ̂²_iid(s,a) + Σ_i p̂_i² ζ̂²(s'_i), where ζ̂²_iid is the ordinary sample variance of augmented path returns and p̂_i is the empirical transition frequency. The correction term adds back the variance of the successor value estimates, weighted by squared transition probabilities—precisely the component that survives in the limit and that the i.i.d. formula omits. To make the estimator practical, the paper derives a node-local statistics scheme (Eqs. 14–17) that keeps per-successor reward mean, reward variance, and transition counts, and reconstructs the same Q̂ and sample variance as the exact stored-history co

What would settle it

Construct a small finite-horizon MDP with one state, one action, a deterministic transition to a single successor, and random rewards. Compute the true replication variance of Q̂ at the root by many independent full-tree runs, and compare it with (a) the i.i.d. sample variance formula and (b) the recursive estimator as N grows. If the i.i.d. estimate's bias does not converge to the variance of the successor value estimate, or if the recursive estimate's scaled error N(ζ̂²−ζ²) does not converge in probability to zero, the paper's central consistency claim is false.

Watch

Extended reading notes

Core claim

The paper's central claim is that the sample variance estimator built on i.i.d. path returns, ζ̂²_iid(s,a) in Eq. (3), is biased for the variance of the Q-function estimator Q̂(s,a), with the bias converging to Σ p_i² Var(V̂(s'_i)) rather than zero when successor value estimates are uncertain (Theorem 2). In its place the paper defines a recursive estimator, ζ̂²(s,a) = ζ̂²_iid(s,a) + Σ p̂_i² ζ̂²(s'_i) (Eq. 12), and proves (Theorem 5) that under convergence of the value estimates, vanishing successor variance, and convergence of the successor variance estimates at a suitable rate, N(s,a)(ζ̂²(s,a) − ζ²(s,a)) converges in probability to zero. The supporting decomposition (Theorem 1) separates t

Load-bearing premise

The load-bearing assumption is that the successor value estimators V̂(s'_i) are independent of each other and of the current node's transition and reward samples; the paper itself notes that online MCTS intertwines the samples, so if the real correlation structure differs from that idealization, the recursive correction may be miscalibrated and the consistency proof does not directly cover the implemented algorithm.

Editorial extensions

If this is right

If the recursive estimator is correct, MCTS selection rules that consume variance estimates (OCBA and UCB1-Tuned) will allocate samples according to the true variance of Q̂, raising the empirical probability of selecting the optimal first action at a fixed simulation budget.
Because the estimator is recursive, variance information propagates through the tree period by period; the bias correction does not fade as the horizon grows, which matters for long-horizon problems like the H=10 inventory and kidney-donation tests.
The node-local implementation makes the consistent estimator usable inside online backpropagation loops without storing full path histories or recomputing from scratch, so the consistency result is not just asymptotic but implementable.
The i.i.d. estimator's bias is nonzero in the limit whenever successor value estimates carry uncertainty, so any variance-aware tree policy built on the old estimator will systematically under-explore actions whose successors are still uncertain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the consistency proof relies on an idealized two-stage sampling setup in which successor value estimates are independent of current-node samples; in a live MCTS the samples are shared, so a direct comparison of the recursive estimator against an exact stored-history computation under the identical sample stream would reveal whether the correction stays calibrated outside the i
Editorial inference: the same correction principle—adding the probability-weighted variance of the bootstrapped target—could apply to other reinforcement-learning estimators in which a shared value estimate enters many samples, e.g., off-policy value learning, where the target's own uncertainty is usually ignored.
Editorial inference: the recursive variance could drive adaptive branching or early stopping decisions; a node whose variance is dominated by successor uncertainty is a candidate for deeper expansion rather than additional samples at the current node, a direction the paper mentions as future work.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper identifies a real bias in the i.i.d. sample variance for Q-estimators in finite-horizon MCTS and gives an exact, inexpensive node-local update for that variance component. But the central claim—that the recursive estimator is consistent—is not supported. Eq. (12) replaces Var(V̂(s')), the variance of a max over actions, with Var(Q̂(s',a*)) at the empirical best action. Those are not equal in general, and no argument connects them. So Theorem 5's assumption (iii) is not satisfied by the implemented estimator; the consistency proof assumes successor variance estimates converge to the quantity the theorem must deliver.

What's solid: Theorem 1's decomposition is correct under the stated two-stage independence, and the authors are upfront that online MCTS violates it. Theorem 2's bias limit is right. The node-local statistics of Section IV—storing reward mean/variance per successor and reconstructing the sample variance—is genuinely useful and provably equivalent to the full-history update. That part is clean.

Soft spots, in proportion: the main one is the consistency gap above. A second: the numerical section deliberately tracks only the local component N ζ̂²_iid, not the recursive ζ̂², so the experiments cannot validate the headline claim. They do show the node-local update removes the bias in the local component, and PCS improves, but the improvement could be coming from the node-local update rather than the recursive correction. A third, acknowledged by the authors: the independence assumption is violated in the actual algorithm, so the decomposition describes an idealized scheme.

If the authors redefine ζ̂²(s') to estimate Var(V̂(s'))—or prove convergence of the max estimator's variance—and report the full recursive estimator in experiments, this becomes a solid paper. As it stands it is a good heuristic with a clean partial theory.

Recommendation: send to peer review. The core idea is worth a serious referee, but expect major revision. I would want to see the consistency claim either proven or explicitly downgraded.

Referee Report

4 major / 4 minor

Summary. The paper studies variance estimation for Q-function estimators in finite-horizon MDP tree search. It derives a variance decomposition for \hat Q(s,a) (Theorem 1), shows that the standard i.i.d. sample variance is asymptotically biased (Theorems 2 and 3), and introduces a recursive variance estimator (Eq. (12)) claimed to be consistent (Theorem 5). To make the estimator practical, the paper develops a node-local statistics update that is shown to be algebraically equivalent to exact updating (Theorem 6), and integrates the estimator into OCBA and UCB1-Tuned selection rules. Numerical experiments on inventory control and kidney paired donation problems report improved empirical probability of correct selection relative to a baseline that uses the approximate variance estimator.

Significance. If fully established, the variance decomposition and a genuinely consistent recursive estimator would be valuable contributions to variance-aware MCTS, particularly for OCBA-style sample allocation. The node-local implementation (Theorem 6) is a strong and verifiable computational contribution: it exactly reproduces the full-history update with per-node statistics. The numerical results are suggestive and the paper is clearly written. However, the central theoretical claim—consistency of the implemented estimator—is not currently supported, because the recursive definition substitutes a different quantity from the one appearing in the variance decomposition, and the consistency proof relies on an unverified rate condition. The paper's own caveats in Sections II-B, V, and VI acknowledge parts of this gap, but the abstract and conclusion state the consistency result without those qualifications.

major comments (4)

[Section III-B, Eqs. (12a)-(12d)] The recursive estimator sets \hat\zeta^2(s'_i) = \hat\zeta^2(s'_i, a^*) with a^* = argmax Q(s'_i, a) (Eqs. (12c)-(12d)). However, Theorem 1's decomposition (Eq. (4)) requires Var(\hat V(s'_i)), where \hat V(s'_i) = max_a \hat Q(s'_i,a). The variance of a maximum of correlated estimators is not generally equal to the variance of the estimator at the empirical best action; the max operator introduces covariance terms and randomness of the argmax. No condition or argument is provided under which the two coincide. Consequently, \hat\zeta^2(s,a) as implemented is not an estimator of the \zeta^2(s,a) defined in Eq. (2), and Theorem 5's conclusion does not apply to the algorithm used in Section V. The numerical section does not remedy this: Section VI explicitly measures only the local component N(s_0,a^*)\hat\zeta^2_{iid}(s_0,a^*), not the recursive \hat\zeta^2(s_0,a^*).
[Theorem 5, assumption (iii)] The consistency proof assumes a rate condition, N(s,a) \sum_i |\hat p_i^2 \hat\zeta^2(s'_i) - p_i^2 Var(\hat V(s'_i))| \xrightarrow{p} 0, which is essentially the inductive conclusion one would need at successor nodes. The update rules in Algorithm 1 do not guarantee this rate: sample counts N(s',i) are random and may be of the same order as N(s,a), so the scaled difference need not vanish from the recurrence alone. The paragraph following Theorem 5 acknowledges that the condition can fail if successor estimates are noisy, but the paper does not supply a separate proof that the proposed updates satisfy (iii). Thus the central consistency claim is conditional on an unverified assumption that is as strong as the theorem itself.
[Section II-B, paragraph before Theorem 1] The variance decomposition assumes that the successor estimators {\hat V(s'_i)} are independent across i and jointly independent of the current-period transitions and rewards. The paper correctly states that this is an 'idealized two-stage sampling scheme' and that in online MCTS the samples used to update \hat V and \hat Q are intertwined. Because both the decomposition (4) and the recursive estimator (12) rely on this assumption, the unqualified statement in the abstract that the proposed estimator is consistent is not supported for the actual algorithms in Section V. The manuscript should either develop a formal analysis for the shared-sample setting or explicitly restrict the consistency claim to the idealized scheme; the current presentation leaves the practical claim unproven.
[Theorem 4] The statement of Theorem 4 is internally inconsistent as written. It says 'the true mean, variance, and transition probability are used, i.e., \hat V(s'_i)=E[\hat V(s'_i)], \hat\zeta^2(s'_i)=Var(\hat V(s'_i)), \hat p_i=p_i.' If \hat V(s'_i) is set equal to its expectation, then the estimator is deterministic and Var(\hat V(s'_i))=0, contradicting the nonzero successor variance appearing in the same sentence and in the conclusion. The proof's invocation of Theorem 3 also conditions on a realization of \hat V, not on its mean. The intended meaning is presumably that the estimator correctly uses the population moments, but as written the theorem and its proof are not coherent and should be rewritten.

minor comments (4)

[Theorems 2 and 3] The notation '− →' is garbled; it should be a standard convergence arrow (e.g., '\xrightarrow{p}' or '\to'). Please correct throughout.
[Section IV-C, example] In the random-transition example of Fig. 4, the values used for \hat V(s_1) and \hat V(s_2) (implicitly 1 and 2) are not stated. The reader must infer them from the computation \hat Q = 0.5(1.5+1) + 0.5(3.5+2). Please make these values explicit.
[Section VI] The variance-estimation error metric is defined as the MAPE of N(s_0,a^*)\hat\zeta^2_{iid}(s_0,a^*) — i.e., the local component. The paper should state more prominently that this metric does not evaluate the recursive estimator \hat\zeta^2(s,a), and that the empirical improvements in PCS do not by themselves validate the consistency claim.
[Author biographies] The author biographies contain 'PLACE PHOTO HERE' placeholders; these should be removed before submission.

Circularity Check

1 steps flagged · score 6.0 of 10

Recursive consistency proof assumes at successor nodes exactly the convergence it is supposed to establish, and Eq. (12c) substitutes a single-action Q-variance for the variance of the max.

self definitional [Section III-B, Eq. (12c)-(12d) and Theorem 5, assumption (iii)]
"ˆζ²(s,a) = ˆζ²_iid(s,a) + Σ_i ˆp_i² ˆζ²(s'_i), (12a) ... (12c) ˆζ²(s'_i) = ˆζ²(s'_i, a*), (12d) a* = arg max_{a'} ˆQ(s'_i, a') ... Theorem 5: iii) the successor states variance estimates converge at a rate satisfying N(s,a)Σ_i |ˆp_i² ˆζ²(s'_i) − p_i² Var(ˆV(s'_i))| \xrightarrow{p} 0."

The variance term that must be propagated from Theorem 1 is Var(ˆV(s'_i)) = Var(max_a ˆQ(s'_i,a)). The implemented estimator (12c)-(12d) instead uses ˆζ²(s'_i,a*) with a* the empirical maximizer. The variance of a data-selected maximum is not the same as the variance estimator for a single fixed action, and the paper provides no argument connecting them. Theorem 5 then assumes in (iii) that the successor estimate already converges to Var(ˆV(s'_i)) at a rate involving N(s,a). That is effectively the same consistency property the recursive estimator is supposed to deliver one level deeper, so the proof imports its conclusion as an assumption rather than deriving it. The central consistency claim is therefore not established from the decomposition alone.

full rationale

The first-principles part of the paper—Theorem 1's variance decomposition and Theorems 2-3 on the asymptotic bias of ˆζ²_iid—is an independent application of the law of total variance under stated independence assumptions; it does not fit parameters and does not rely on load-bearing self-citation. However, the recursively defined correction is not derived as a closed chain. Equations (12c)-(12d) define the successor variance proxy as ˆζ²(s'_i,a*) at the empirical maximizer, while the Bellman variance recursion requires Var(ˆV(s'_i)) = Var(max_a ˆQ(s'_i,a)). The variance of a data-selected maximum differs from the variance estimator for a single action; no proof addresses the argmax selection or correlations among actions. Theorem 5 then proves consistency of ˆζ²(s,a) only under assumption (iii), which requires exactly that the successor estimates converge to Var(ˆV(s'_i)) with a rate scaled by N(s,a). This is essentially the same convergence property the theorem is meant to establish at the successor level, and the rate condition is not derived from successor sample sizes. Thus the central claim of a consistent recursive estimator is not self-contained: its proof assumes at successor nodes the very convergence it claims to prove. The numerical section reinforces the gap by reporting only the local component N(s0,a*)ˆζ²_iid(s0,a*) rather than the recursive ˆζ²(s0,a*) itself, so the experiments do not close the missing link. No fitted-parameter or self-citation circularity is present; the circular structure is in the recursive consistency proof.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central derivation uses standard probability tools; there are no fitted constants in the variance formula itself. The free parameters are algorithmic hyperparameters (warm-up, epsilon). The load-bearing assumptions are the independence of value estimates, the unproven convergence-rate condition for consistency, and the implicit variance-of-max proxy. These are what the consistency claim rests on beyond the published literature.

free parameters (2)

warm-up sample count n_0(s) = 4 for root node, 2 for all other nodes (Sec. V-C)
Chosen by hand to ensure enough samples before applying variance-based selection. It affects the finite-sample behavior of the algorithm but is not fitted to the theoretical claim.
normalization floor ε in periodwise range W(t) = not specified; defined as max(Qmax(t)-Qmin(t), ε) (Sec. V-B)
Introduced to avoid division by zero in UCB1-Tuned normalization. Its value is not reported and is an ad hoc numerical safeguard.

assumptions (5)

standard math Bellman optimality equations for finite-horizon MDPs
Used throughout Section II to define Q and V and to justify the recursive estimators (1a)-(1b).
standard math Law of total variance and conditional covariance formulas
Used in the proof of Theorem 1 to decompose Var(\hat Q). This is textbook probability.
domain assumption Independence of successor value estimators \hat V(s'_i) across i and from current-period samples
Stated before Theorem 1; the paper acknowledges this fails in online MCTS because samples are shared/intertwined. The recursive estimator is built on this assumption.
ad hoc to paper Rate condition in Theorem 5(iii): N(s,a) Σ |\hat p_i² \hat ζ²(s'_i) - p_i² Var(\hat V(s'_i))| → 0
Introduced specifically to make the consistency proof work. It is not proven to hold for the MCTS allocation rules, and the paper itself warns that error can accumulate in deep trees.
ad hoc to paper Var(\hat V(s')) ≈ Var(\hat Q(s', a*)) where a* = argmax \hat Q
Used in Eq. (12c)-(12d) without derivation. The maximum of correlated estimators has additional variance from the max operator and from cross-action correlations; the paper treats the variance of the empirical best action as the variance of the state value.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Consistent Variance Estimation for Q-Function Estimators in Finite-Horizon MDP Tree Search." pith.science (2026). https://pith.science/paper/VMFIOTOA

@misc{pith2026260714555,
  author       = {Pith},
  title        = {Pith review of: Consistent Variance Estimation for Q-Function Estimators in Finite-Horizon MDP Tree Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VMFIOTOA}},
  note         = {Machine review of arXiv:2607.14555}
}

read the original abstract

We study the variance of Q-function estimators in finite-horizon, finite-state Markov decision process (MDP) tree search. We show that the variance decomposes into three components attributed to the immediate reward collected, probabilistic state transitions, and uncertainty in future state value function estimates. Using this decomposition, we show that the sample variance estimator based on the assumption of i.i.d. paths is biased, underestimating the true variance, and the bias does not vanish in the limit. We then propose a recursive variance estimator that is consistent. To enable efficient storage and computation, we derive an equivalent implementation of the recursive estimator using only node-local statistics that can be iteratively updated. This consistent variance estimator is integrated into two Monte Carlo Tree Search (MCTS) sampling procedures for finite-horizon MDPs. In numerical examples from inventory control and kidney paired donation matching, the new estimator improves the performance of the MCTS algorithm relative to a baseline that uses the i.i.d.-based sample variance estimator.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 1 linked inside Pith

[1]

R. S. Sutton and A. G. Barto,Reinforcement Learning: An Introduction. MIT Press, 2018

2018
[2]

Bertsekas and J

D. Bertsekas and J. N. Tsitsiklis,Neuro-Dynamic Programming. Athena Scientific, 1996

1996
[3]

W. B. Powell,Approximate Dynamic Programming: Solving the Curses of Dimensionality. John Wiley & Sons, 2007. 13

2007
[4]

Finite-time analysis of the multiarmed bandit problem,

P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,”Machine Learning, vol. 47, pp. 235–256, 2002

2002
[5]

Bandit based Monte Carlo planning,

L. Kocsis and C. Szepesv ´ari, “Bandit based Monte Carlo planning,” in European Conference on Machine Learning. Springer, 2006, pp. 282– 293

2006
[6]

Minimax regret bounds for reinforcement learning,

M. G. Azar, I. Osband, and R. Munos, “Minimax regret bounds for reinforcement learning,” inProceedings of the 34th International Conference on Machine Learning (ICML), vol. 70. PMLR, 2017, pp. 263–272. [Online]. Available: https://proceedings.mlr.press/v70/azar17a. html

2017
[7]

A survey of Monte Carlo tree search methods,

C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton, “A survey of Monte Carlo tree search methods,”IEEE Transactions on Computational Intelligence and AI in Games, vol. 4, no. 1, pp. 1–43, 2012

2012
[8]

A tutorial for Monte Carlo tree search in AI,

M. C. Fu, D. Qiu, and J. Xu, “A tutorial for Monte Carlo tree search in AI,” in2024 Winter Simulation Conference (WSC). IEEE, 2024, pp. 16–30

2024

Show all 20 references

[9]

Monte Carlo tree search with optimal com- puting budget allocation,

Y . Li, M. Fu, and J. Xu, “Monte Carlo tree search with optimal com- puting budget allocation,” in2019 IEEE 58th Conference on Decision and Control (CDC). IEEE, 2019, pp. 6332–6337

2019
[10]

An optimal computing budget allocation tree policy for Monte Carlo tree search,

Y . Li, M. C. Fu, and J. Xu, “An optimal computing budget allocation tree policy for Monte Carlo tree search,”IEEE Transactions on Automatic Control, vol. 67, no. 6, pp. 2685–2699, 2022

2022
[11]

The variance of discounted Markov decision processes,

M. J. Sobel, “The variance of discounted Markov decision processes,” Journal of Applied Probability, vol. 19, no. 4, pp. 794–802, 1982. [Online]. Available: http://www.jstor.org/stable/3213832

1982
[12]

Mean-variance tradeoffs in an undiscounted MDP,

——, “Mean-variance tradeoffs in an undiscounted MDP,”Operations Research, vol. 42, no. 1, pp. 175–183, 1994, accessed 28 July 2025. [Online]. Available: http://www.jstor.org/stable/171533

1994
[13]

Mean-variance optimization in Markov decision processes,

S. Mannor and J. Tsitsiklis, “Mean-variance optimization in Markov decision processes,” https://arxiv.org/abs/1104.5601, 2011, a full version of an ICML 2011 paper

2011 arXiv
[14]

Efficient selectivity and backup operators in Monte Carlo tree search,

R. Coulom, “Efficient selectivity and backup operators in Monte Carlo tree search,” inInternational Conference on Computers and Games. Springer, 2006, pp. 72–83

2006
[15]

Monte Carlo tree search: A tutorial,

M. C. Fu, “Monte Carlo tree search: A tutorial,” in2018 Winter Simulation Conference (WSC). IEEE, 2018, pp. 222–236

2018
[16]

Simulation-based algorithms for Markov decision processes: Monte Carlo tree search from AlphaGo to AlphaZero,

——, “Simulation-based algorithms for Markov decision processes: Monte Carlo tree search from AlphaGo to AlphaZero,”Asia-Pacific Journal of Operational Research, vol. 36, no. 06, p. 1940009, 2019

2019
[17]

Mastering the game of Go without human knowledge,

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, and A. Bolton, “Mastering the game of Go without human knowledge,”Nature, vol. 550, no. 7676, pp. 354–359, 2017

2017
[18]

A kidney paired donation program simulation,

Z. Yue, M. C. Fu, H. El-Amine, J. Xu, and C.-H. Chen, “A kidney paired donation program simulation,” in2024 Winter Simulation Conference (WSC). IEEE, 2024, pp. 1011–1022. Zhenyu Yuereceived the bachelor’s degree in mathematics and computer science from Uni- versity of Maryland...

2024
[2014]

He is the author of two books, including a best seller: Stochas- tic Simulation Optimization: An Optimal Computing Budget Allocation (World Scientific, 2011). Prof. Chen has served as a Department Editor for IIE Transactions and Asia–Pacific Journal of Operational Research, an...

2011
[2018]

candidate of applied math at University of Maryland, College Park, MD, USA

He is a Ph.D. candidate of applied math at University of Maryland, College Park, MD, USA. His research interests include stochastic processes and optimization. PLACE PHOTO HERE Jie Xu(Senior Member, IEEE) received the B.S. degree in electrical engineering from Nanjing Universi...

1999

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

R. S. Sutton and A. G. Barto,Reinforcement Learning: An Introduction. MIT Press, 2018

2018

[2] [2]

Bertsekas and J

D. Bertsekas and J. N. Tsitsiklis,Neuro-Dynamic Programming. Athena Scientific, 1996

1996

[3] [3]

W. B. Powell,Approximate Dynamic Programming: Solving the Curses of Dimensionality. John Wiley & Sons, 2007. 13

2007

[4] [4]

Finite-time analysis of the multiarmed bandit problem,

P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,”Machine Learning, vol. 47, pp. 235–256, 2002

2002

[5] [5]

Bandit based Monte Carlo planning,

L. Kocsis and C. Szepesv ´ari, “Bandit based Monte Carlo planning,” in European Conference on Machine Learning. Springer, 2006, pp. 282– 293

2006

[6] [6]

Minimax regret bounds for reinforcement learning,

M. G. Azar, I. Osband, and R. Munos, “Minimax regret bounds for reinforcement learning,” inProceedings of the 34th International Conference on Machine Learning (ICML), vol. 70. PMLR, 2017, pp. 263–272. [Online]. Available: https://proceedings.mlr.press/v70/azar17a. html

2017

[7] [7]

A survey of Monte Carlo tree search methods,

C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton, “A survey of Monte Carlo tree search methods,”IEEE Transactions on Computational Intelligence and AI in Games, vol. 4, no. 1, pp. 1–43, 2012

2012

[8] [8]

A tutorial for Monte Carlo tree search in AI,

M. C. Fu, D. Qiu, and J. Xu, “A tutorial for Monte Carlo tree search in AI,” in2024 Winter Simulation Conference (WSC). IEEE, 2024, pp. 16–30

2024