REVIEW 2 major objections 4 minor 30 references
Spectral Analysis of Dueling Q-Learning
T0 review · 2 major / 4 minor · reviewed 2026-07-10 · grok-4.5
Pith's one-line read Unregularized dueling Q-learning converges when value and advantage gains make the joint spectral radius of its switching modes less than one.
desk verdict Solid, carefully written extension of SLS/JSR analysis to unregularized centered dueling Q-learning; the math holds under an explicit JSR premise, novelty is moderate, scope is tabular/i.i.d. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The switching family A^{VA}(α,β) (and its block counterpart B^{VA}) obtained by writing the Bellman residual in common/differential coordinates: the joint spectral radius of this family is the single scalar that certifies both exponential stability of the deterministic recursion and the drift term needed for the stochastic finite-time bound.
What would settle it
Construct a small finite MDP, choose gains α,β that violate the paper’s ℓ∞ contraction interval, compute the joint spectral radius of the resulting mode family, and check whether the deterministic recursion still converges; if it diverges while the radius is greater than one, the spectral certificate is sharp.
Extended reading notes
Core claim
Centered tabular dueling Q-learning with constant gains α and β is equivalent to the Q-space recursion Q_{k+1}=Q_k+(|A|αΠ+βΠ⊥)D(F(Q_k)−Q_k). Its error is an exact switching linear system whose modes belong to the convex hull of the deterministic-policy family A^{VA}(α,β). Whenever the joint spectral radius of that family is less than one, the deterministic trajectory converges exponentially to Q⋆ and the sampled recursion admits an explicit finite-time first-moment bound whose residual radius vanishes with √α.
Load-bearing premise
The whole convergence argument rests on the joint spectral radius of the gain-dependent mode family being strictly less than one—a condition that is hard to check for a general MDP and is only given explicitly under a conservative common-norm bound.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes unregularized, unprojected tabular dueling Q-learning with constant step-sizes. It interprets the update via the orthogonal common/differential decomposition Q = ΠQ + Π⊥Q, with separate gains α (value/common) and β (advantage/differential), and shows that the deterministic error recursion is exactly a switching linear system whose modes form the family A^{VA}(α,β) (or the equivalent block family B^{VA}). Under the joint spectral radius condition ρ(A^{VA}(α,β)) < 1, deterministic iterates converge exponentially to Q* (Prop. 5.1 / Lem. 5.5). For i.i.d. sampling, Theorem 6.1 gives a finite-time first-moment bound with an O(√α) neighborhood of Q*. The analysis also supplies a conservative ℓ_∞ gain interval (Lem. 5.9), equivalence of the AV and Q-only forms (Lem. 5.6), and a one-state example that makes the rate separation explicit.
Significance. If the results hold, the paper fills a clear gap left by Daley et al.: convergence for the pure centered tabular recursion without regularization or projection, together with an exact SLS representation and a finite-time stochastic bound. The orthogonal decomposition and two-gain preconditioner give a transparent control-theoretic reading of how value and advantage streams act on action-common versus action-differential components. Strengths include careful equivalence of coordinate systems (Lem. 5.6, 5.8), an explicit conservative gain condition that makes the JSR hypothesis hold, and a fully written martingale-difference + product-bound argument in Appendix D. The main limitation is that the theorems are conditional on a family JSR that is hard to check for general MDPs, and the stochastic analysis is restricted to full-support i.i.d. sampling.
major comments (2)
- The central deterministic and stochastic guarantees (Prop. 5.1, Lem. 5.5, Thm. 6.1) are conditional on ρ(A^{VA}(α,β)) < 1 (or the stricter common ℓ_∞ contraction of Lem. 5.9). That premise is stated explicitly and is not hidden, but for general MDPs JSR is hard to verify; the paper only supplies a conservative gain interval and a one-state example. A short discussion of when the condition is expected to hold, or of practical checks (e.g., small MDPs, numerical JSR bounds), would strengthen the claim that the unregularized recursion is usable beyond the special cases already treated.
- Theorem 6.1 and the surrounding development treat only full-support i.i.d. sampling. The introduction and §3 note that the same conditional-mean/noise decomposition can be combined with Markovian SA arguments as in the author’s prior work, but that extension is not carried out. For a journal contribution on stochastic RL, either a brief Markovian extension or a clearer statement of the limitation as future work would better match the scope of the finite-time claim.
minor comments (4)
- In §5.1–5.2 the notation V and A is used both for the centered components of Q and (in related work) for conventional value/advantage functions; a one-sentence reminder that they are component coordinates unless stated otherwise would reduce possible confusion.
- Figures 1 and 2 are informative but the captions could state the exact gain choices and the plotted norm (ℓ2 vs ℓ∞) more explicitly so the comparison is self-contained.
- A few typographical issues remain (e.g., “un known”, “Q-fu nction” in the abstract; occasional spacing in arXiv line breaks). A light copy-edit pass would help.
- The relation to Baird’s advantage updating and to Tang et al. VA-learning is discussed in §2; a short table or bullet list of algorithmic differences (centered vs uncentered storage, one vs two gains, regularization) would make the positioning easier to scan.
Circularity Check
No significant circularity: SLS/JSR framework is reused from the author's prior Q-learning work, but the dueling modes, centered decomposition, and finite-time bound are derived rather than assumed.
-
self citation load bearing
[Sec. 3.2–3.3, Lem. 3.1–3.2; also Lem. 4.1–4.2 and citations to [12]]
"The following finite-family piecewise-quadratic construction [6, 12] is the Lyapunov certificate used in the deterministic arguments. ... The rest of the paper uses this JSR implication as the main convergence certificate. ... The proof can be found in [12]. We omit the details."
The paper's stability certificate (JSR Lyapunov norm and the implication 'JSR<1 ⇒ exponential decay') is imported from the author's prior Q-learning SLS paper [12] rather than re-proved. This is ordinary methodological reuse, not a uniqueness theorem that forbids alternatives, and the dueling-specific modes A^{VA}(α,β) and the finite-time bound are still derived independently. It does not force the main claims by construction.
full rationale
The paper's central claims are conditional theorems of the form 'if ρ(A^{VA}(α,β))<1 then exponential convergence / O(√α) neighborhood,' not free-standing predictions. The exact SLS representation (Lem. 5.2, 5.7), the Π/Π⊥ orthogonal decomposition, the Q-space ↔ AV-space similarity (Lem. 5.8), the conservative ℓ∞ gain interval (Lem. 5.9), and the martingale product-bound argument for Thm. 6.1 are all derived from the algorithm definition and standard Bellman selector lemmas. The only self-citation of substance is the author's prior SLS/JSR toolkit for ordinary Q-learning (arXiv:2604.19569), which supplies the Lyapunov certificate (Lem. 3.1–3.2) and the baseline AQ family; that is ordinary reuse of a method, not a uniqueness theorem or fitted constant that forces the dueling result. No parameter is fitted to data and then relabeled a prediction; step-sizes remain free design parameters under an explicit stability hypothesis. Score 1 for minor framework self-citation that is not load-bearing for the new content.
Assumptions & free parameters
free parameters (2)
- step-size α (value/common gain)
- step-size β (advantage/differential gain)
assumptions (6)
- domain assumption Finite state/action discounted MDP with γ∈(0,1) and bounded rewards
- domain assumption Sampling distribution d has full support: d(s,a)>0 for all (s,a)
- domain assumption i.i.d. state-action samples for the stochastic finite-time bound
- ad hoc to paper Joint spectral radius of the dueling mode family is less than one (or the ℓ_∞ gain condition of Lem. 5.9)
- standard math Bellman optimality operator has unique fixed point Q*; policy-dependent linearization of F via measurable selectors
- standard math JSR Lyapunov construction and convex-hull invariance of JSR for finite families
invented entities (1)
-
Centered AV coordinates (V=ΠQ, A=Π^⊥Q) with two-gain preconditioner C^{VA}
independent evidence
Cite this review
Pith. "Pith review of Spectral Analysis of Dueling Q-Learning." pith.science (2026). https://pith.science/paper/3IV2AFDT
@misc{pith2026260708340,
author = {Pith},
title = {Pith review of: Spectral Analysis of Dueling Q-Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/3IV2AFDT}},
note = {Machine review of arXiv:2607.08340}
}
read the original abstract
Q-learning is a fundamental algorithm in reinforcement learning (RL) for solving discounted Markov decision processes (MDPs) when the transition kernel is unknown. The deep Q-network (DQN) extends Q-learning by using a deep neural network for Q-function approximation, which makes Q-learning applicable to more practical high-dimensional problems. Dueling Q-learning decomposes the Q-function into a value function and an advantage function and learns the two components jointly, which can improve learning efficiency. However, the theoretical understanding of dueling Q-learning is still limited. Recent work has initiated an analysis of tabular dueling Q-learning, but existing guarantees focus on a regularized formulation and leave the pure tabular update less completely understood. This paper strengthens that line of analysis by adding a direct interpretation of the centered tabular decomposition and by establishing convergence guarantees for the unregularized, unprojected constant step-size recursion. In particular, we derive an exact switching linear system representation for deterministic dueling Q-learning and a finite-time error bound in expectation for the sampled stochastic version. The analysis clarifies how the value and advantage updates act as different gains on the action-common (value function) and action-differential (advantage function) components of the Q-function.
Figures
Reference graph
Works this paper leans on
-
[1]
Gian-Carlo Rota and W. Gilbert Strang. A note on the joint spectral radius. Indagationes Mathematicae, 22:379–381, 1960; also in Proceedings of the Koninklijke Nederlandse Akademie van Wetenschappen, Series A , 63:379–381, 1960. doi:10.1016/S1385-7258(60)50046-1
-
[2]
Rapha¨ el M. Jungers. The Joint Spectral Radius: Theory and Applications . Lecture Notes in Control and Information Sciences, volume 385. Springer, Berlin and Heidelberg, 2009. doi:10.1007/978-3-540-95980-9
-
[3]
Switching in Systems and Control
Daniel Liberzon. Switching in Systems and Control . Systems & Control: Foundations & Applications. Birkh¨ auser Boston, Boston, MA, 2003. doi:10.1007/978-1-4612-0017-8
-
[4]
Hai Lin and Panos J. Antsaklis. Stability and stabilizab ility of switched linear systems: A survey of recent results. IEEE Transactions on Automatic Control , 54(2):308–322, 2009. doi:10.1109/TAC.2008.2012009
-
[5]
Sta- 31 bility criteria for switched and hybrid systems
Robert Shorten, Fabian Wirth, Oliver Mason, Kai Wulff, and Christopher King. Sta- 31 bility criteria for switched and hybrid systems. SIAM Review , 49(4):545–592, 2007. doi:10.1137/05063516X
-
[6]
Generating fun ctions of switched linear systems: Analysis, computation, and stability applications
Jianghai Hu, Jinglai Shen, and Wei Zhang. Generating fun ctions of switched linear systems: Analysis, computation, and stability applications. IEEE Transactions on Automatic Control , 56(5):1059–1074, 2011. doi:10.1109/TAC.2010.2067590
-
[7]
Christopher J. C. H. Watkins and Peter Dayan. Q-learning . Machine Learning , 8:279–292,
-
[8]
doi:10.1007/BF00992698
Show all 30 references
-
[9]
Puterman
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Pr ogramming. Wiley Series in Probability and Statistics. Wiley, New York , 1994. doi:10.1002/9780470316887
1994 doi
-
[10]
Bertsekas and John N
Dimitri P. Bertsekas and John N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA, 1996
1996
-
[11]
Sutton and Andrew G
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . Second edition. MIT Press, Cambridge, MA, 2018
2018
-
[12]
Maguluri, Sanjay Shakkottai, and K arthikeyan Shanmugam
Zaiwei Chen, Siva T. Maguluri, Sanjay Shakkottai, and K arthikeyan Shanmugam. A Lyapunov theory for finite-sample guarantees of Markovian stochasti c approximation. Operations Re- search, 72(4):1352–1367, 2024. Published online October 6, 2023. doi:10.1287/opre.2022.0249
2024 doi
- [13]
-
[14]
Brett Daley, Prabhat Nagarajan, Martha White, and Marl os C. Machado. An analysis of action-value temporal-difference methods that l earn state values. Re- inforcement Learning Journal , 6:2619–2636, 2025. Presented at the Reinforce- ment Learning Conference, Edmonton, Alberta,...
2025
-
[15]
Rusu, Joel Veness, Marc G
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andr ei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidj eland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, H elen King, Dharshan Kumaran, Daan Wiers...
2015 doi
-
[16]
Dueling network architectures for deep re inforcement learn- ing
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. Dueling network architectures for deep re inforcement learn- ing. In Proceedings of the 33rd International Conference on Machine L earning, Pro- ceedings of Machine Learning Research, v...
2003
-
[17]
Baird III
Leemon C. Baird III. Reinforcement learning in continu ous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Ne tworks (ICNN’94), volume 4, pages 2448–2453. IEEE, 1994. doi:10.1109/ICNN.1994.374604
1994 doi
-
[18]
Wiering and Hado van Hasselt
Marco A. Wiering and Hado van Hasselt. Two novel on-poli cy reinforcement learning algo- rithms based on TD( λ)-methods. In Proceedings of the IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learni ng (ADPRL) , pages 280–287,
-
[19]
doi:10.1109/ADPRL.2007.368200. 32
2007 doi
-
[20]
Wiering and Hado van Hasselt
Marco A. Wiering and Hado van Hasselt. The QV family comp ared to other reinforcement learning algorithms. In 2009 IEEE Symposium on Adaptive Dynamic Programming and Rein - forcement Learning, pages 101–108. IEEE, 2009. doi:10.1109/ADPRL.2009.4927532
2009 doi
-
[21]
V A-learning as a more efficient alternative to Q-learning
Yunhao Tang, R´ emi Munos, Mark Rowland, and Michal Valko. V A-learning as a more efficient alternative to Q-learning. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, volume 202, pag es 33739–33757. PMLR,
-
[22]
33 Appendix A Auxiliary switching-system and block-operator facts Lemma A.1
https://proceedings.mlr.press/v202/tang23h.html. 33 Appendix A Auxiliary switching-system and block-operator facts Lemma A.1. For a finite matrix family H ={A1, . . . , A M}⊂ Rm× m, one has ρ(co(H)) = ρ(H), where the left-hand side is computed over all convex combina tions of m...
-
[23]
holds. Then, there exist a norm p on Rn and a constant c > 0, independent of the step-size α , with c¯α < 1, such that for every 0 < α ≤ ¯α , p(M x)≤ (1− cα )p(x), ∀x∈ Rn, ∀M∈ co(AV A(α, β )), (66) wheneverAV A(α, β ) is formed with the fixed-ratio gains in (32). Proof. Write ¯...
-
[24]
(67) Choose θ such that ρ(AV A( ¯α, ¯β )) < θ < 1
formed with α = ¯α and β = ¯β ; explicitly, ¯AV A π := I− ( |A|¯α Π + ¯β Π ⊥ ) D(I− γP Γ π ). (67) Choose θ such that ρ(AV A( ¯α, ¯β )) < θ < 1. By Lemma 3.1 , there is a norm p such that p( ¯AV A π x)≤ θp(x), x ∈ Rn, π ∈ Θ . By convexity of the norm, the same bound holds for ...
-
[25]
This gives ( 70)
and ( 32), the two modes differ only through the Bellman policy selecto r: AV A µ − AV A ν = αγ C V A α DP (Γ µ− Γ ν ). This gives ( 70). The supremum in ( 69) is finite because the stochastic-policy set is a finite produ ct of probability simplices and is compact, while the norm...
-
[26]
Fix a stochastic policy ¯µ :S→ ∆ |A|
(73) Proof. Fix a stochastic policy ¯µ :S→ ∆ |A|. Let yk+1 = AV A ¯µ yk + αξ k, y 0 = Q0− Q⋆, and decompose yk = ¯yk + ˜yk, where ¯yk+1 = AV A ¯µ ¯yk, ¯y0 = Q0− Q⋆, ˜yk+1 = AV A ¯µ ˜yk + αξ k, ˜y0 = 0. Let ek := Qk− Q⋆− yk. Subtracting the fixed-policy filter from (
-
[27]
gives e0 = 0 and ek+1 = (Qk+1− Q⋆)− yk+1 = AV A µ k (Qk− Q⋆) + αξ k− ( AV A ¯µ yk + αξ k ) = AV A µ k ek + (AV A µ k − AV A ¯µ )yk = AV A µ k ek + (AV A µ k − AV A ¯µ )¯yk + (AV A µ k − AV A ¯µ )˜yk. Accordingly, write ek = uk + vk, where the two parts are generated by the two...
-
[28]
The last step follows by the change of variables j = k− 1− t and the geometric bound k− 1∑ t=0 (1− cα )2(k− 1− t) = k− 1∑ j=0 (1− cα )2j≤ 1 1− (1− cα )2≤ 1 cα , 0 < cα < 1
and ( 71) imply, for k≥ 1, E[∥˜yk∥2 2] = α 2 k− 1∑ t=0 E [ (AV A ¯µ )k− 1− tξt 2 2 ] ≤ α 2K 2 k− 1∑ t=0 (1− cα )2(k− 1− t)E[∥ξt∥2 2] ≤ α 2K 2 k− 1∑ t=0 (1− cα )2(k− 1− t){σ 2 0 + σ 2 1Mk− 1} ≤ αK 2 c {σ 2 0 + σ 2 1Mk− 1}. The last step follows by the change of variab...
-
[29]
Summing over t = 0,
and ( 70), each summand satisfies AV A µ k− 1···AV A µ t+1(AV A µ t − AV A ¯µ )¯yt 2 ≤ AV A µ k− 1···AV A µ t+1 2 AV A µ t − AV A ¯µ 2∥¯yt∥2 ≤ K(1− cα )k− 1− t αγL ∥¯yt∥2 ≤ K(1− cα )k− 1− t αγL K (1− cα )t∥Q0− Q⋆∥2 . Summing over t = 0, . . . , k − ...
-
[30]
For 0 < α ≤ α 0, one has C1ασ 2 1≤ 1/ 2 and Mk− 1≤ Mk
Taking the supremum over 0 ≤ r ≤ k and using the definitions of C0 and C1 gives Mk≤ { 1 + 4K 2 + 4 (γLK 2 c )2} ∥Q0− Q⋆∥2 2 + 4α (K 2 c + γ2L2K 4 c3 ) {σ 2 0 + σ 2 1Mk− 1} = C0∥Q0− Q⋆∥2 2 + C1α{σ 2 0 + σ 2 1Mk− 1}. For 0 < α ≤ α 0, one has C1ασ 2 1≤ 1/ 2 and Mk− 1≤ Mk. Hence Mk...
Reviewed July 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.