Pith. sign in

REVIEW 3 major objections 4 minor 27 references

An Analysis of Action-Value Temporal-Difference Methods That Learn State Values

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Bootstrapping from a learned state value can be sound, and in the dueling family it is also faster: the paper proves QV-learning converges and introduces RDQ, which beats Dueling DQN on all five MinAtar games.

desk verdict Solid theory on QV-learning; the RDQ deep RL claim is plausible but undersupported because the MinAtar comparison is confounded and the minimum-norm mechanism is not actually tested. read the letter →

arxiv 2507.09523 v2 pith:7VUTX3OD submitted 2025-07-13 cs.LG cs.AI

Brett Daley , Prabhat Nagarajan , Martha White , Marlos C. Machado This is my paper
classification cs.LGcs.AI
keywords TDlearningQV-learningAV-learningduelingarchitecturesadvantageestimationregularizedQ-learningconvergenceanalysisMinAtar
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks when it pays to learn a state-value function alongside an action-value function in temporal-difference reinforcement learning, instead of learning only $Q(s,a)$. It separates such algorithms into QV-learning, where the two functions bootstrap from each other, and AV-learning, where the action value is decomposed as $Q(s,a)=V(s)+\mathrm{Adv}(s,a)$. The authors prove that the expected QV-learning update is a contraction whose unique fixed point is the correct on-policy pair, show that the standard QV off-policy variant QVMAX is biased, and introduce a corrected version. They then propose Regularized Dueling Q-learning (RDQ), which replaces the mean-advantage subtraction of dueling networks with an $\ell^2$ penalty, and report that RDQ significantly outperforms Dueling DQN on all five MinAtar games with the same network architecture and hyperparameters.

What carries the argument

At the center of the convergence result is the joint QV operator $H([q;v]) = b + A[q;v]$ with $b=[r; E_b r]$ and $A=\gamma [0\ P;\ 0\ E_b P]$. Because every row of $A$ is a probability distribution scaled by $\gamma$, the operator norm is $\|A\|_\infty = \gamma$, so $H$ is a maximum-norm contraction and, by the fixed-point theorem, has unique fixed point $[q_b; v_b]$. At the center of the AV-learning result is the decomposition $Q(s,a)=V(s)+\mathrm{Adv}(s,a)$ together with the $\ell^2$ penalty $\frac{1}{2}V(s)^2 + \frac{1}{2}\sum_a \mathrm{Adv}(s,a)^2$, which replaces the mean-advantage subtraction of the original dueling network. The $\ell^2$ penalty selects the point on the solution line closest to zero, the minimum-norm solution, and in the tabular setting the penalty-free invariant $V(s)=\sum_a \mathrm{Adv}(s,a)$ gives the same geometric mechanism for Hard RDQ.

What would settle it

Sweep $\beta$ over a wide range in the same MinAtar setup while keeping Dueling DQN's architecture and tuned hyperparameters fixed; if some $\beta$ lets Dueling DQN match or exceed RDQ, the claim that RDQ significantly outperforms it is weakened. In addition, a tabular MDP with a known non-greedy behavior policy should show QVMAX asymptoting to a biased fixed point with $\|q^*-Q\|$ bounded away from zero, as Proposition 3.2 predicts.

Watch

Extended reading notes

Core claim

The paper's central claim is that learning a state-value function as an intermediate step in learning action values can be sound and useful, but the two families of methods behave differently. For QV-learning, the expected update is an affine joint operator $H([q;v]) = b + A[q;v]$, with $b=[r; E_b r]$ and $A=\gamma [0\ P;\ 0\ E_b P]$, and since $\|A\|_\infty=\gamma<1$, the operator is a contraction whose unique fixed point is the on-policy pair $[q_b; v_b]$; this is the paper's proof that both value estimates progress together. For off-policy control it shows QVMAX is biased because its $V$ update conditions on the state only, so $[q^*; v^*]$ is not a fixed point, and it gives a corrected version, BC-QVMAX, whose unique fixed point is $[q^*; v^*]$ but which behaves like a lagged Q-learning. The AV-learning analysis derives a tabular version of Dueling Q-learning and shows that subtracting the mean advantage is only one way to make the decomposition identifiable; RDQ instead uses an $\ell^2$ penalty on $V(s)$ and $\mathrm{Adv}(s,a)$ to find a minimum-norm solution of $Q(s,a)=V(s)+\mathrm{Adv}(s,a)$. In the deep RL setting, RDQ with the same dueling architecture and the same tuned hyperparameters as Dueling DQN outperforms Dueling DQN on all five MinAtar games, supporting the paper's overall finding that state-value learning helps in prediction for both families, but in control only AV-learning delivers major gains over Q-learning.

Load-bearing premise

The MinAtar comparison is clean only if the single untuned regularization strength $\beta=10^{-3}$ is suitable for all five games and if the tuned DQN and Dueling DQN baselines were not disadvantaged by the implementation; otherwise the reported advantage could come from tuning rather than from the algorithmic change.

Editorial extensions

If this is right

  • QV-learning converges to the on-policy pair $(q_b, v_b)$ even though $V$ never needs to converge first; the joint operator is a contraction, so both value estimates make progress together.
  • QVMAX is biased for off-policy control: its state-value update is conditioned only on the state, so its fixed point is not $(q^*, v^*)$, and any use of QVMAX inherits this bias.
  • BC-QVMAX restores the correct fixed point $(q^*, v^*)$, but its smoothing makes it essentially a delayed, learned approximation of Q-learning, so it does not deliver a sample-efficiency benefit over Q-learning.
  • In the AV-learning family, dueling methods can substantially outperform Q-learning in tabular control, and Hard RDQ's minimum-norm path is at least as efficient as mean-advantage subtraction.
  • In the MinAtar deep RL benchmark, RDQ with the same dueling architecture and the same tuned hyperparameters as Dueling DQN achieves higher mean scores in all five games.
  • Because RDQ's $\ell^2$ penalty is added to the loss rather than hard-coded into the network, the same regularization can be combined with other dueling-style losses without changing the network architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The success of RDQ suggests that the dueling architecture's benefit comes less from estimating true advantages than from providing a lower-dimensional parameterization, so any penalty that anchors $V$ and $\mathrm{Adv}$ near zero may produce similar gains.
  • The QV-learning contraction proof likely extends to other asymmetric pairings of value functions, since the argument only requires one function to serve as a bootstrap target under a fixed behavior policy.
  • A per-game sweep of the regularization strength $\beta$ would clarify whether Dueling DQN's mean-advantage subtraction is actively harmful or merely different; the paper reports only one untuned value of $\beta$.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper studies temporal-difference methods that learn both state-value and action-value functions, dividing them into QV-learning (directly updating Q and V) and AV-learning (decomposing Q into V plus an advantage function). The main theoretical results are: (i) Theorem 3.1, showing that the expected QV-learning update is an affine contraction with fixed point [q_b; v_b]; (ii) Proposition 3.2, showing that QVMAX is biased in off-policy control; and (iii) Proposition 3.3, giving the unique fixed point of a newly proposed bias-corrected variant, BC-QVMAX. The paper then formalizes tabular Dueling Q-learning and introduces Regularized Dueling Q-learning (RDQ), which replaces the mean-advantage identifiability subtraction with an l2 penalty; in MinAtar experiments, Soft RDQ is reported to significantly outperform DQN and Dueling DQN under the same network architecture.

Significance. The theoretical analysis is clean and fills a real gap: to my knowledge, no prior published convergence proof for QV-learning exists, and the bias of QVMAX had not been identified. The tabular experiments are careful, with step-size sweeps and confidence intervals, and the code is publicly available. The RDQ idea is interesting and the paper provides a clear geometric motivation. However, the deep RL result, which is the paper's headline empirical claim, is compromised by a confounded comparison: Soft RDQ differs from Dueling DQN in both the output decomposition and the presence of an l2 penalty, and no ablation isolates the mechanism. As a result, the support for the minimum-norm explanation is weak, though the algorithm itself may still be valuable.

major comments (3)
  1. [Section 4.3, Eq. (12), Figure 6] The MinAtar comparison of Soft RDQ against Dueling DQN changes two factors at once: Q(s,a) is computed without the mean-advantage subtraction (Eq. 11 versus Eq. 8), and the loss includes the l2 penalty of Eq. (12). The l2 penalty is a generic output regularizer, so the reported gains cannot be attributed to the minimum-norm mechanism without additional controls. The tabular Hard RDQ, which removes the mean term but has no penalty, only slightly outperforms Dueling Q-learning in a small majority of cases (Figure 5, right), indicating that removing the identifiability term alone does not explain the large deep RL improvements. The authors should add ablations: (a) Dueling DQN with the same l2 penalty added to its loss, (b) Soft RDQ with the mean-advantage term retained, and (c) a sensitivity analysis over beta. This is necessary to support the paper's central claim that RDQ's advantage comes from the proposed minimum-norm decomposition rather than from generic output regularization.
  2. [Abstract] The abstract states that 'both families are more efficient than Expected Sarsa in the prediction setting,' but the prediction-setting experiment in Section 3.1 evaluates QV-learning only; no prediction experiment for an AV-learning method is presented. This overstates the evidence. Please either add a prediction experiment for an AV-learning method or revise the abstract to say that QV-learning was found to be more efficient than Expected Sarsa in prediction, and that the control experiments showed benefits for AV-learning.
  3. [Section 4.2, Eqs. (13)-(14)] The tabular Soft RDQ update is inconsistent with the loss in Eq. (12). With the penalty coefficient beta in the loss and step size alpha, ordinary gradient descent would give a decay factor of (1 - alpha * beta), not (1 - beta). Since alpha is typically much smaller than 1, the two formulations behave very differently. This does not affect the Hard RDQ (beta = 0) invariant or the deep RL implementation, which uses the stated loss with Adam, but the derivation as written is unclear and should be corrected or clarified.
minor comments (4)
  1. [Section 4.3] The phrase 'significantly outperforms' is used without a formal statistical test; consider reporting a paired test or explicitly discussing the overlap of the 95% confidence intervals at the evaluation points.
  2. [Figure 1 (right)] The claim that QV-learning 'scales better' rests on a visual comparison of slopes; consider reporting the fitted slopes or a statistical comparison to make the claim quantitative.
  3. [Section 4.2, Eq. (15)] The invariant V(s) = sum_a Adv(s,a) holds only up to the initialization difference V0(s) - sum_a Adv0(s,a); this is stated in the footnote but should be mentioned in the main text for clarity.
  4. [Appendix A, Theorem 3.1 proof] The proof relies on operator notation from Daley (2025), a PhD thesis; since the notation is defined in the appendix, the paper is self-contained, but the reliance on a non-archival reference is slightly unusual and worth noting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the theoretical results are self-contained operator calculations and the RDQ experiments are genuine comparisons; the only self-citation is notational and not load-bearing.

full rationale

The paper's load-bearing claims are derived independently rather than assumed. Theorem 3.1 (QV-learning contraction) is proven in Appendix A.1 directly from the operator definitions and the identity v_b = E_b q_b; the proof explicitly verifies the fixed point and computes ||Hy - Hy'||_∞ ≤ γ||y - y'||_∞, so the contraction claim is not a restatement of its conclusion. Propositions 3.2 and 3.3 are direct operator computations showing where the behavior-policy operator E_b enters and why BC-QVMAX's interleaved updates unroll to the Bellman optimality operator; these are not circular. The RDQ algorithm is introduced as a new loss (Eqs. 11-14), and its tabular minimum-norm property is derived from the invariant in Eq. 15; the deep RL experiments then compare Soft RDQ against DQN and Dueling DQN with stated hyperparameters and an untuned β = 10^{-3}. Even if the β penalty could be viewed as a confound with generic output regularization, that is an experimental-design concern, not a reduction of the claimed outcome to the paper's own inputs by construction. The only self-citation, the operator convention of Daley (2025) in Appendix A, is purely notational and fully defined in Table 1, so it is not load-bearing. There is no fitted parameter being renamed as a prediction and no uniqueness theorem imported from the authors' prior work. The claims are therefore self-contained and no circularity is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The free parameters are standard hyperparameters: beta for RDQ regularization and step sizes. The theoretical results rely on standard MDP/operator assumptions. No invented entities.

free parameters (2)
  • beta (regularization coefficient for RDQ) = 1e-3
    The regularization strength for the l2 penalty in Eq. (12). The paper states 'We choose beta = 10^-3 for the regularization strength; we did not tune this value.' It is a hyperparameter chosen by hand, not fitted to the test data.
  • Step sizes for tabular experiments = chosen by grid search over 61 values
    The step size alpha for the tabular experiments was selected by minimizing AUC over a grid. This is standard hyperparameter selection, not a free parameter in the theoretical claims.
assumptions (3)
  • standard math Standard Bellman operator properties for finite MDPs
    The paper assumes standard finite MDP definitions and uses Banach fixed-point theorem. This is standard background.
  • domain assumption The QV-learning updates with the same step size α and the order of Q before V
    The convergence proof relies on the specific update order and shared step size from Wiering (2005). The paper states 'it is commonly assumed that both updates share the same step size, α, as is shown above.' This is a modeling assumption.
  • domain assumption MinAtar games are representative of RL benchmarks
    The Deep RL claims are based on five MinAtar games. Generalization to other domains is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Analysis of Action-Value Temporal-Difference Methods That Learn State Values." pith.science (2026). https://pith.science/paper/7VUTX3OD

@misc{pith2026250709523,
  author       = {Pith},
  title        = {Pith review of: An Analysis of Action-Value Temporal-Difference Methods That Learn State Values},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VUTX3OD}},
  note         = {Machine review of arXiv:2507.09523}
}
read the original abstract

The hallmark feature of temporal-difference (TD) learning is bootstrapping: using value predictions to generate new value predictions. The vast majority of TD methods for control learn a policy by bootstrapping from a single action-value function (e.g., Q-learning and Sarsa). Significantly less attention has been given to methods that bootstrap from two asymmetric value functions: i.e., methods that learn state values as an intermediate step in learning action values. Existing algorithms in this vein can be categorized as either QV-learning or AV-learning. Though these algorithms have been investigated to some degree in prior work, it remains unclear if and when it is advantageous to learn two value functions instead of just one -- and whether such approaches are theoretically sound in general. In this paper, we analyze these algorithmic families in terms of convergence and sample efficiency. We find that while both families are more efficient than Expected Sarsa in the prediction setting, only AV-learning methods offer any major benefit over Q-learning in the control setting. Finally, we introduce a new AV-learning algorithm called Regularized Dueling Q-learning (RDQ), which significantly outperforms Dueling DQN in the MinAtar benchmark.

Figures

Figures reproduced from arXiv: 2507.09523 by the authors.

Figure 1
Figure 1. On-policy prediction performance of QV-learning compared to Expected Sarsa in the [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Depiction of bootstrapping in QV￾learning variants. The arrows point from the bootstrapped value function to the value function being updated. One consequence of this change is that now there is reciprocal bootstrapping between Q and V . Pre￾viously, with QV-learning, we had only a unidirec￾tional information flow: V bootstrapped from itself, and Q bootstrapped from V . Hence, Q could not corrupt the state values in… view at source ↗
Figure 3
Figure 3. Off-policy control performance of BC-QVMAX compared to QVMAX and Q-learning [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: RDQ finds the minimum-norm solu￾tion to the underdetermined system of equations Q(s, a) = V (s) + Adv(s, a), ∀a ∈ A. Thus, RDQ may require fewer updates than Dueling Q￾learning to converge in practice. We propose a new dueling algorithm which does not rely on subtracti…
Figure 5
Figure 5. Figure 5: Off-policy control performance of Hard RDQ compared to Dueling Q-learning and Q [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Deep RL results for five MinAtar games. Averaged across 30 trials; shading represents [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 24 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Leemon C. Baird. Advantage Updating . Technical Report WL-TR-93‑1146, Wright-Patterson Air Force Base, 1993

    work page 1993

  3. [3]

    Sur les op \'e rations dans les ensembles abstraits et leur application aux \'e quations int \'e grales

    Stefan Banach. Sur les op \'e rations dans les ensembles abstraits et leur application aux \'e quations int \'e grales. Fundamenta Mathematicae, 3 0 (1): 0 133--181, 1922

    work page 1922

  4. [4]

    Dynamic Programming

    Richard Bellman. Dynamic Programming. Princeton University Press, 1957

    1957

  5. [5]

    Bertsekas and John N

    Dimitri P. Bertsekas and John N. Tsitsiklis. Neuro-Dynamic Programming . Athena Scientific, 1996

    work page 1996

  6. [6]

    Multistep Credit Assignment in Deep Reinforcement Learning

    Brett Daley. Multistep Credit Assignment in Deep Reinforcement Learning. PhD thesis, University of Alberta, 2025

    work page 2025

  7. [7]

    ChainerRL : A deep reinforcement learning library

    Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa. ChainerRL : A deep reinforcement learning library. Journal of Machine Learning Research, 22 0 (77): 0 1--14, 2021

    work page 2021

  8. [8]

    George H. John. When the best move isn't optimal: Q-learning with exploration. In AAAI Conference on Artificial Intelligence (AAAI), 1994

    work page 1994

Show all 27 references
  1. [9]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam : A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015

    2015

  2. [10]

    Orr, and Klaus-Robert M \"u ller

    Yann LeCun, L \'e on Bottou, Genevieve B. Orr, and Klaus-Robert M \"u ller. Efficient BackProp . In Neural Networks: Tricks of the Trade. Springer, 2002

    2002

  3. [11]

    Rusu, Joel Veness, Marc G

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518 0 (7540): 0 529--533, 2015

    2015

  4. [12]

    Towards model-free RL algorithms that scale well with unstructured data, 2023

    Joseph Modayil and Zaheer Abbas. Towards model-free RL algorithms that scale well with unstructured data, 2023. arXiv:2311.02215

    2023 arXiv

  5. [13]

    Revisiting Rainbow : Promoting more insightful and inclusive deep reinforcement learning research

    Johan Samir Obando-Ceron and Pablo Samuel Castro. Revisiting Rainbow : Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning (ICML), 2021

    2021

  6. [14]

    Rummery and Mahesan Niranjan

    Gavin A. Rummery and Mahesan Niranjan. On-line Q-learning using connectionist systems. Technical Report CUED/F-INFENG/TR 166, University of Cambridge, 1994

    1994

  7. [15]

    Matthia Sabatelli, Gilles Louppe, Pierre Geurts, and Marco A. Wiering. The deep quality-value family of deep reinforcement learning algorithms. In International Joint Conference on Neural Networks (IJCNN), 2020

    2020

  8. [16]

    Littman, and Csaba Szepesv \'a ri

    Satinder Singh, Tommi Jaakkola, Michael L. Littman, and Csaba Szepesv \'a ri. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine Learning, 38: 0 287--308, 2000

    2000

  9. [17]

    Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3: 0 9--44, 1988

    1988

  10. [18]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018

    2018

  11. [19]

    VA-learning as a more efficient alternative to Q-learning

    Yunhao Tang, R \'e mi Munos, Mark Rowland, and Michal Valko. VA-learning as a more efficient alternative to Q-learning . In International Conference on Machine Learning (ICML), 2023

    2023

  12. [20]

    Double Q-learning

    Hado van Hasselt. Double Q-learning . In Neural Information Processing Systems (NeurIPS), 2010

    2010

  13. [21]

    Insights in Reinforcement Learning

    Hado van Hasselt. Insights in Reinforcement Learning. PhD thesis, Utrecht University, 2011

    2011

  14. [22]

    Dueling network architectures for deep reinforcement learning

    Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning (ICML), 2016

    2016

  15. [23]

    Christopher J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, University of Cambridge, 1989

    1989

  16. [24]

    Marco A. Wiering. QV( )-learning : A new on-policy reinforcement learning algorithm. In European Workshop on Reinforcement Learning (EWRL), 2005

    2005

  17. [25]

    Wiering and Hado van Hasselt

    Marco A. Wiering and Hado van Hasselt. Two novel on-policy reinforcement learning algorithms based on TD( )-methods . In IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), 2007

    2007

  18. [26]

    Wiering and Hado van Hasselt

    Marco A. Wiering and Hado van Hasselt. The QV family compared to other reinforcement learning algorithms. In IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), 2009

    2009

  19. [27]

    MinAtar : An Atari -inspired testbed for thorough and reproducible reinforcement learning experiments, 2019

    Kenny Young and Tian Tian. MinAtar : An Atari -inspired testbed for thorough and reproducible reinforcement learning experiments, 2019. arXiv:1903.03176

    2019 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.