{"id":"25fe3c80-f34f-4691-84b5-d8e2425b303a","arxiv_id":"2607.08340","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"Unregularized centered dueling Q-learning converges under a joint spectral radius condition, with value and advantage acting as distinct gains on common and differential Q components.","lead":"This paper proves that unregularized tabular dueling Q-learning is an exact switching linear system whose value and advantage updates apply different gains to the action-common and action-differential parts of Q. That gives a joint-spectral-radius convergence certificate and a finite-time first-moment error bound for constant step-sizes under i.i.d. sampling.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged JSR premise.","rationale":"The reader's weakest_assumption correctly isolates the governing JSR condition and the i.i.d.-only scope of the stochastic bound. Those are genuine practical limitations, but they are already treated as explicit premises in the theorems; they do not constitute an internal flaw that would force a stronger rejection. The mathematical development is careful, the equivalence of the AV-space and Q-space recursions is checked both ways (Lem. 5.6), and the finite-time argument follows the standard pattern of the author's earlier Q-learning SLS work. No additional load-bearing gap (e.g., an unjustified interchange of max and expectation, a non-measurable selector, or a missing projection step) appears. Therefore the CONDITIONAL verdict stands: accept the claims under the stated JSR/gain hypothesis, with the same caveats on verifiability and sampling model already noted by the reader.","tokens_in":34698,"tokens_out":552,"duration_ms":6147,"concrete_test":"Independently recompute the two deterministic-policy modes of Example 5.1, form the 2×2 products of length up to 4, and verify that the numerical JSR of A^{VA}(α,β) equals max{|1-α(1-γ)|,|1-β/2|} for the stated (α,β,γ); if the equality fails, the simultaneous-triangularization claim (and therefore the rate-separation illustration) is wrong.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims (exact SLS form of the centered unregularized recursion, Prop. 5.1 / Lem. 5.5 exponential convergence under ρ(A^{VA}(α,β))<1, and Thm. 6.1 O(√α) first-moment neighborhood under i.i.d. sampling) are internally consistent. The proofs correctly use the orthogonal decomposition Π/Π⊥, the measurable Bellman selector of Lem. 3.4, the similarity of block and Q-space families (Lem. 5.8), and a standard martingale-difference + product-bound argument. The only load-bearing premise is already named by the reader: the family JSR (or the stricter common ℓ∞ contraction of Lem. 5.9). That premise is stated explicitly as a hypothesis, not hidden; the paper supplies a conservative gain interval that makes it hold and a one-state example that makes the rate separation transparent. No internal inconsistency or missing step that would invalidate the theorems as written was found.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper analyzes unregularized, unprojected tabular dueling Q-learning with constant step-sizes. It interprets the update via the orthogonal common/differential decomposition Q = ΠQ + Π⊥Q, with separate gains α (value/common) and β (advantage/differential), and shows that the deterministic error recursion is exactly a switching linear system whose modes form the family A^{VA}(α,β) (or the equivalent block family B^{VA}). Under the joint spectral radius condition ρ(A^{VA}(α,β)) < 1, deterministic iterates converge exponentially to Q* (Prop. 5.1 / Lem. 5.5). For i.i.d. sampling, Theorem 6.1 gives a finite-time first-moment bound with an O(√α) neighborhood of Q*. The analysis also supplies a conservative ℓ_∞ gain interval (Lem. 5.9), equivalence of the AV and Q-only forms (Lem. 5.6), and a one-state example that makes the rate separation explicit.","tokens_in":34994,"tokens_out":1038,"duration_ms":9547,"significance":"If the results hold, the paper fills a clear gap left by Daley et al.: convergence for the pure centered tabular recursion without regularization or projection, together with an exact SLS representation and a finite-time stochastic bound. The orthogonal decomposition and two-gain preconditioner give a transparent control-theoretic reading of how value and advantage streams act on action-common versus action-differential components. Strengths include careful equivalence of coordinate systems (Lem. 5.6, 5.8), an explicit conservative gain condition that makes the JSR hypothesis hold, and a fully written martingale-difference + product-bound argument in Appendix D. The main limitation is that the theorems are conditional on a family JSR that is hard to check for general MDPs, and the stochastic analysis is restricted to full-support i.i.d. sampling.","major_comments":[{"comment":"The central deterministic and stochastic guarantees (Prop. 5.1, Lem. 5.5, Thm. 6.1) are conditional on ρ(A^{VA}(α,β)) < 1 (or the stricter common ℓ_∞ contraction of Lem. 5.9). That premise is stated explicitly and is not hidden, but for general MDPs JSR is hard to verify; the paper only supplies a conservative gain interval and a one-state example. A short discussion of when the condition is expected to hold, or of practical checks (e.g., small MDPs, numerical JSR bounds), would strengthen the claim that the unregularized recursion is usable beyond the special cases already treated.","section":null},{"comment":"Theorem 6.1 and the surrounding development treat only full-support i.i.d. sampling. The introduction and §3 note that the same conditional-mean/noise decomposition can be combined with Markovian SA arguments as in the author’s prior work, but that extension is not carried out. For a journal contribution on stochastic RL, either a brief Markovian extension or a clearer statement of the limitation as future work would better match the scope of the finite-time claim.","section":null}],"minor_comments":[{"comment":"In §5.1–5.2 the notation V and A is used both for the centered components of Q and (in related work) for conventional value/advantage functions; a one-sentence reminder that they are component coordinates unless stated otherwise would reduce possible confusion.","section":null},{"comment":"Figures 1 and 2 are informative but the captions could state the exact gain choices and the plotted norm (ℓ2 vs ℓ∞) more explicitly so the comparison is self-contained.","section":null},{"comment":"A few typographical issues remain (e.g., “un known”, “Q-fu nction” in the abstract; occasional spacing in arXiv line breaks). A light copy-edit pass would help.","section":null},{"comment":"The relation to Baird’s advantage updating and to Tang et al. VA-learning is discussed in §2; a short table or bullet list of algorithmic differences (centered vs uncentered storage, one vs two gains, regularization) would make the positioning easier to scan.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The manuscript is a solid, carefully written extension of the author’s SLS/JSR line for Q-learning to the centered dueling case. Novelty relative to Daley et al. is real but incremental (unregularized analysis + finite-time bound + SLS interpretation). Fit for a theory-oriented RL/control journal is good; for a broader ML venue the limited empirical content and the i.i.d.-only stochastic result may be a concern. I see no integrity or citation issues."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This paper closes a clean gap: unregularized, unprojected constant-step tabular dueling Q-learning, written as an exact switching linear system with a two-gain preconditioner on the action-common and action-differential subspaces, plus a finite-time first-moment bound under i.i.d. sampling.\n\nWhat is new is the centered decomposition V=ΠQ, A=Π⊥Q with C^{VA}=|A|αΠ+βΠ⊥, the exact block/Q-space similarity (so the JSRs match), the deterministic SLS and Prop. 5.1 / Lem. 5.5 under ρ(A^{VA}(α,β))<1, and Thm. 6.1’s O(√α) neighborhood. It sits on Daley et al. (regularized AV) and the author’s prior SLS treatment of standard Q-learning, but the unregularized centered case and the two-gain rate separation were not done. The proofs are careful: orthogonal projections, measurable Bellman selectors, martingale noise, product bounds. The one-state example makes the common-vs-differential gain effect transparent, and Lem. 5.9 gives a usable conservative ℓ∞ gain interval.\n\nSoft spots are real but already flagged and proportionate. The load-bearing premise is the JSR condition (or the stricter common-contraction interval); it is stated as a hypothesis, not hidden, and hard to check on general MDPs. Stochastic analysis is i.i.d. only; Markovian extension is left as a pointer. No code or numerical JSR checks beyond the toy example. None of that breaks the theorems as written.\n\nThis is for people who care about tabular value-based theory and SLS/JSR certificates. Practitioners looking for deep-RL guidance will not find it. The citation pattern is appropriate; the math is internally consistent.\n\nI would send it to peer review. A serious referee can pressure-test the gain conditions and ask for a Markovian remark or a small multi-state check, but the core claims deserve that time. Worth reading if you work in this corner; I would cite the SLS form and the finite-time bound when discussing dueling updates.","headline":"Solid, carefully written extension of SLS/JSR analysis to unregularized centered dueling Q-learning; the math holds under an explicit JSR premise, novelty is moderate, scope is tabular/i.i.d.","tokens_in":35616,"tokens_out":557,"would_cite":true,"duration_ms":24425,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","90C40","93D05"],"pacs":[],"model":"grok-4.5","headline":"Unregularized dueling Q-learning converges when value and advantage gains make the joint spectral radius of its switching modes less than one.","keywords":["dueling Q-learning","switching linear systems","joint spectral radius","centered decomposition","constant step-size","finite-time error bound","tabular RL","value-advantage separation"],"falsifier":"Construct a small finite MDP, choose gains α,β that violate the paper’s ℓ∞ contraction interval, compute the joint spectral radius of the resulting mode family, and check whether the deterministic recursion still converges; if it diverges while the radius is greater than one, the spectral certificate is sharp.","tokens_in":35559,"feed_emoji":"⚙️","tokens_out":732,"duration_ms":38907,"temperature":0.7,"pith_summary":"Dueling Q-learning splits the action-value table into a state-common piece and a centered action-differential piece, then multiplies the Bellman residual by two different gains on those pieces. Earlier theory covered only regularized or projected versions of that idea. This paper shows that the pure, unregularized, constant-step recursion is exactly a switching linear system whose modes are the usual Bellman operators preconditioned by those two gains. If the joint spectral radius of the mode family is strictly less than one, the deterministic iterates converge exponentially to the optimal Q-function, and the i.i.d. sampled version stays inside an O(√α) first-moment neighborhood of that optimum. The same spectral picture explains why a larger gain on the common component can accelerate learning relative to ordinary Q-learning without changing the differential gain.","feed_headline":"Dueling Q-learning converges when its gain modes have radius <1","feed_subtitle":"Value and advantage act as separate gains; spectral radius certifies both deterministic and sampled rates.","key_machinery":"The switching family A^{VA}(α,β) (and its block counterpart B^{VA}) obtained by writing the Bellman residual in common/differential coordinates: the joint spectral radius of this family is the single scalar that certifies both exponential stability of the deterministic recursion and the drift term needed for the stochastic finite-time bound.","core_discovery":"Centered tabular dueling Q-learning with constant gains α and β is equivalent to the Q-space recursion Q_{k+1}=Q_k+(|A|αΠ+βΠ⊥)D(F(Q_k)−Q_k). Its error is an exact switching linear system whose modes belong to the convex hull of the deterministic-policy family A^{VA}(α,β). Whenever the joint spectral radius of that family is less than one, the deterministic trajectory converges exponentially to Q⋆ and the sampled recursion admits an explicit finite-time first-moment bound whose residual radius vanishes with √α.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Dueling Q-learning is a switching linear system controlled by spectral radius","Centered tabular dueling Q yields exact modes from value-advantage gains","Joint spectral radius <1 certifies deterministic and sampled dueling rates","Value and advantage act as gains: radius of A^{VA} family ensures convergence","Unregularized constant-step dueling Q admits finite-time first-moment bound"],"cache_read_input_tokens":17664,"weakest_assumption_plain":"The whole convergence argument rests on the joint spectral radius of the gain-dependent mode family being strictly less than one—a condition that is hard to check for a general MDP and is only given explicitly under a conservative common-norm bound.","fun_headline_variants_meta":{"raw":{"variants":["Dueling Q-learning is a switching linear system controlled by spectral radius","Centered tabular dueling Q yields exact modes from value-advantage gains","Joint spectral radius <1 certifies deterministic and sampled dueling rates","Value and advantage act as gains: radius of A^{VA} family ensures convergence","Unregularized constant-step dueling Q admits finite-time first-moment bound"]},"model":"grok-4.5","effort":"low","cost_usd":0.004438,"raw_usage":{"total_tokens":1353,"prompt_tokens":824,"num_sources_used":0,"completion_tokens":101,"cost_in_usd_ticks":44380000,"prompt_tokens_details":{"text_tokens":824,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":428,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":824,"tokens_out":101,"duration_ms":4278,"temperature":1.0,"reasoning_tokens":428,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T09:04:40.948776+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Construct a small finite MDP, choose gains α,β that violate the paper’s ℓ∞ contraction interval, compute the joint spectral radius of the resulting mode family, and check whether the deterministic recursion still converges; if it diverges while the radius is greater than one, the spectral certificate is sharp.","supporting_citations":[],"review_version":1}