REVIEW 4 major objections 4 minor 21 references
rQdia: Regularizing Q-Value Distributions With Image Augmentation
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read rQdia claims that a single auxiliary loss—mean-squared-error equalization of Q-value distributions between an image and its random-shift augmentation over the mini-batch's actions—improves sample efficiency and final performance in…
desk verdict A simple, plausible idea for regularizing Q-value distributions, buried under an unfinished draft with self-contradictory Atari numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The paper's central identity is that for a state $s$ and its augmentation $\mathrm{aug}(s)$, the Q-value distribution should be preserved: $Q(s,a_i)\approx Q(\mathrm{aug}(s),a_i)$ for actions $a_i$ in the mini-batch. The enforcement mechanism is the auxiliary loss $$L_{\mathrm{rQdia}}=\frac{1}{n}\sum_{i<n}\left(Q(s,a_i)-Q(\mathrm{aug}(s),a_i)\right)^2,$$ added to the agent's standard loss. The augmentation is a 4-pixel pad followed by a random inward crop, i.e., a translation. The mini-batch actions stand in for a distribution of realistic actions, so the loss does not need ground-truth Q-labels for unselected actions. In Rainbow's discrete-action setting the loss runs over the full action space, and the paper reports that MSE between Q-value logits worked better than KL divergence between distributional probabilities.
What would settle it
Run the 100k MuJoCo suite with rQdia using actions sampled on-policy instead of mini-batch actions, holding all other hyperparameters fixed; if the improvement disappears or flips, the paper's claim that mini-batch actions serve as a good proxy for the action distribution is the mechanism. On Atari, compare rQdia-MSE against rQdia-KL on the same 26 games; the paper's appendix already shows the two order differently on PrivateEye, so a full sign-reversal across games would test whether the equalization form or the invariance itself drives the results.
Extended reading notes
Core claim
The central discovery is that Q-value distributions carry a trainable invariance signal: for a given state, the vector of Q-values across actions should not change when the input image is perturbed by a translation. rQdia enforces this by adding the auxiliary loss to the agent's normal loss, computing the mean squared error between Q(s, a_i) and Q(aug(s), a_i) over the actions in the mini-batch. Because labels are only available for actions actually taken, the paper uses all actions in the mini-batch to form the distribution, treating them as 'randomly sampled historical actions'. In Rainbow's discrete-action setting, the same loss is applied over the full action space and directly regularizes the action-value distribution. The reported outcome is that this simple equalization improves DrQ/SAC on 9/12 and 10/12 MuJoCo tasks and Data-Efficient Rainbow on 18/26 Atari environments, with larger gains early in training.
Load-bearing premise
The load-bearing premise is that matching Q-values between an image and its augmentation for the actions that happen to be in the mini-batch sufficiently captures the full Q-value distribution, and that forcing this invariance is what causes the reported gains.
Editorial extensions
If this is right
- On the DeepMind Control Suite from pixels, rQdia improves DrQ's sample efficiency and final scores at both 100k and 500k steps, and the rQdia+DrQ combination beats the state-embedding SAC baseline on more tasks and by wider margins than DrQ or CURL alone.
- On the 100k-step Atari benchmark, rQdia added to Data-Efficient Rainbow improves mean episode reward on 18/26 environments and outperforms CURL's contrastive approach on 15/26.
- The loss is an add-on, not an architectural change: it can be appended to SAC-style and Rainbow-style agents with a few lines of code, keeping all original hyperparameters.
- In discrete action spaces rQdia regularizes the action distribution directly, and the paper suggests that directly equalizing action log-probabilities in continuous control is an open question.
Reading between the lines
- Editorial inference: if Q-value distribution invariance is the active ingredient, the same loss should transfer to other input perturbations beyond translations, such as color jitter or intensity shifts, on tasks where those transformations preserve reward.
- Editorial inference: the mini-batch action choice is a coverage assumption; comparing it with actions sampled from the current policy or selected by state similarity would tell whether better coverage of the action space strengthens or weakens the effect.
- Editorial inference: the reported failure on finger_spin with DrQ but not SAC-AE suggests the regularizer's benefit interacts with the base agent's exploration or augmentation schedule; ablating augmentation strength and batch composition would map that interaction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes rQdia, an auxiliary loss that penalizes the mean squared error between Q-values computed on an input image and Q-values computed on an augmented (randomly shifted) image, evaluated over the actions present in the current mini-batch. The loss is added to existing RL objectives for continuous control (DrQ/SAC-AE) and discrete control (Data-Efficient Rainbow). The authors claim improved sample efficiency and final performance in the DeepMind Control Suite and Atari 100k benchmark, and report that the method pushes pixel-based DrQ above a state-embedding SAC baseline.
Significance. The core idea is simple, clearly stated, and not circular: rQdia is a regularizer with no constants fitted to the reported results, and its definition in Eq. (1) is unambiguous. If the empirical claims were reproducible, the method would be a useful addition to pixel-based RL. However, the central quantitative claims are not currently assessable: the paper's own tables contradict the headline counts, no code or data are provided, no error bars or per-seed results are given, and several figures and table references are placeholders. The significance of the paper therefore cannot be established from the manuscript as written.
major comments (4)
- [Abstract; §5.3; Table 1] The headline Atari claim is internally inconsistent. The abstract and Table 1's caption state that rQdia improves on Data-Efficient Rainbow in 18/26 environments; §5.3 says improvements over both CURL and Eff. Rainbow on 15/26; Table 1's bottom row reports 16/26 for rQdia. A direct comparison of the rQdia-Rainbow and Eff. Rainbow columns in Table 1 gives 20/26 wins for rQdia (losses only on BankHeist, Breakout, CrazyClimber, Pong, PrivateEye, and RoadRunner). Because the central claim is a count of benchmark improvements, this discrepancy is load-bearing, and the count must be corrected and defined precisely.
- [Abstract; §5.1; §5.2; Table 2] The continuous-control win counts are also not reproducible from the reported numbers. The abstract claims rQdia boosts DrQ and SAC on 9/12 and 10/12 tasks respectively, and §5.1 reports 5/6 in 100k while §5.2 reports 4/6 in 500k. However, in Table 2, DrQ+rQdia beats DrQ in 4/6 tasks at 500k and 4/6 tasks at 100k, for a total of 8/12, not 9/12; versus SAC-AE the totals are 6/6 and 6/6, which is 12/12, not 10/12. The paper should state exactly which baseline and which step-counts define each reported fraction, and should provide per-seed values.
- [§4; Eq. (1); Algorithm 1] The method's distributional interpretation is not supported by the loss as written. Eq. (1) compares Q(st, ai) with Q(aug(st), ai) only for actions ai in the current mini-batch, which are actions paired with other states. The paper equates this to using 'randomly sampled historical actions,' but provides no evidence that these actions constitute a representative sample of the action distribution relevant to state st. For continuous action spaces, applying MSE on a finite set of mini-batch actions does not equalize the full Q-value distributions described in the introduction. Algorithm 1 additionally introduces a min over the two Q-functions and does not specify a stop-gradient on the target, making the exact training objective ambiguous.
- [§6.3; Appendix C; Table 1] The empirical claims are not verifiable from the submitted materials. The paper states in §6.3 that code is submitted with the paper and in Appendix C that code will be released upon notification, but no code or data are actually provided; Table 1 reports only means over 3 seeds with no error bars or per-seed breakdowns; and the Eff. Rainbow scores are copied from prior work with a different number of seeds. Without code, per-seed data, and significance or variance information, none of the claimed improvements can be checked, and the contradictions noted above cannot be resolved.
minor comments (4)
- [Figures 4, 5, 7, 8; §5.4] Figures 4 and 5 have placeholder captions containing 'Lorem ipsum' and 'Caption place holder,' and Figures 7 and 8 are referenced without descriptive captions; §5.4 directs the reader to 'Figure 2' for results, but Figure 2 is an illustrative related-work figure. The manuscript is not in a complete state for review.
- [Algorithm 1] Algorithm 1 uses the symbol a for both the environment action and the mini-batch action, and the rQdia update is written with the min over Q-functions while Eq. (1) uses a single Q-function; the relationship between the two formulations should be clarified, including where gradients are stopped.
- [Table 5; §F] Table 5 omits the Human and Random columns in the middle of the table, shows a partial '/26' entry in the bottom row, and does not state whether the KL loss is applied to logits or to probabilities; these presentation issues make the KL ablation difficult to interpret.
- [References; Appendix A] Some citations are unresolved in the bibliography: Appendix A refers to '[38]' and '[30]' instead of author names, and the reference list contains duplicate entries for Krizhevsky et al. The paper should use consistent citation keys throughout.
Circularity Check
No significant circularity: rQdia's auxiliary loss is an added regularizer with no fitted constants or self-citation load-bearing; claimed gains are external benchmark results.
full rationale
The derivation chain is non-circular. rQdia's only novel component is the auxiliary MSE loss in Equation (1), which penalizes disagreement between Q(s,a) and Q(aug(s),a) over mini-batch actions; this loss is added to the existing SAC/DrQ or Data-Efficient Rainbow training objectives rather than fitted to the reported outcomes. No parameter is tuned to the benchmark scores, no prediction is defined in terms of the target metric, and no load-bearing result is imported from the authors' prior work (the reference list contains no self-citations). The paper's benchmark claims (9/12, 10/12, 18/26) are direct empirical comparisons against published baselines with hyperparameters inherited from those baselines. The internal inconsistency in the Atari win counts (Table 1's bottom row says 16/26, the text says 18/26, and a direct per-game count gives 20/26) is a reproducibility and correctness concern, not a circularity concern, because the counts do not reduce to the method's definitions or to any fitted quantity. Overall score 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Random translation (pad 4 pixels, then crop 4 pixels) is a useful augmentation for both DMControl and Atari.
- ad hoc to paper Mini-batch actions form a representative sample of the action distribution for each state.
- domain assumption Enforcing invariance of Q-value distributions under image augmentation improves visual representations and RL performance.
- domain assumption In Rainbow, the Q-value distribution is equivalent to the action distribution.
Cite this review
Pith. "Pith review of rQdia: Regularizing Q-Value Distributions With Image Augmentation." pith.science (2026). https://pith.science/paper/RMZTUE4Y
@misc{pith2026250621367,
author = {Pith},
title = {Pith review of: rQdia: Regularizing Q-Value Distributions With Image Augmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/RMZTUE4Y}},
note = {Machine review of arXiv:2506.21367}
}
read the original abstract
rQdia regularizes Q-value distributions with augmented images in pixel-based deep reinforcement learning. With a simple auxiliary loss, that equalizes these distributions via MSE, rQdia boosts DrQ and SAC on 9/12 and 10/12 tasks respectively in the MuJoCo Continuous Control Suite from pixels, and Data-Efficient Rainbow on 18/26 Atari Arcade environments. Gains are measured in both sample efficiency and longer-term training. Moreover, the addition of rQdia finally propels model-free continuous control from pixels over the state encoding baseline.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[4]
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
Abhinav Gupta, Adithyavairavan Murali, Dhiraj Gandhi, and Lerrel Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. arXiv preprint arXiv:1807.07049,
-
[5]
Soft actor-critic algorithms and appli- cations
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05905,
-
[6]
Dream to control: Learning behaviors by latent imagination
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019a. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on M...
arXiv 1912
-
[7]
Imagenet classification with deep con- volutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097–1105, 2012a. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processi...
arXiv 1907
-
[8]
Sharada Mohanty, Jyotish Poonganam, Adrien Gaidon, Andrey Kolobov, Blake Wulfe, Dipam Chakraborty, Graˇzvydas ˇSemetulskis, Jo˜ao Schapke, Jonas Kubilius, Jurgis Paˇsukonis, et al. Mea- suring sample efficiency and generalization in reinforcement learning benchmarks: Neurips 2020 procgen benchmark. arXiv preprint arXiv:2103.15332,
arXiv 2020
-
[11]
Auto- matic data augmentation for generalization in deep reinforcement learning
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Auto- matic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006.12862,
arXiv 2006
-
[12]
Prioritized experience replay.arXiv preprint arXiv:1511.05952,
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay.arXiv preprint arXiv:1511.05952,
-
[14]
URL https://arxiv.org/abs/2007.05929. Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1134–1141. IEEE,
arXiv 2007
Show all 21 references
-
[17]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov
URL https://arxiv.org/ abs/2004.04136. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958,
2004 arXiv
-
[19]
Do- main randomization for transferring deep neural networks from simulation to the real world
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pp. 23–...
2017
-
[21]
Dueling network architectures for deep reinforcement learning
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995–2003. PMLR,
1995
-
[84]
Code for continuous control and discrete Atari will be released on GitHub upon notification of decision and is provided separately together with the supplementary material
Replay Buffer Size 100000 Frame Skip 4 Action repeat 4 Q-network Channels 32, 64 Q-network Filter Size 5 × 5, 5 × 5 Q-network Stride 5, 5 Q-network Hidden Units 256 Momentum τ 0.001 Non-Linearity ReLU Reward Clipping [−1, 1] Multi Step Return 20 Min replay size for sampling 16...
2000
-
[1991]
Im- proving sample efficiency in model-free reinforcement learning from images
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741,
1910 arXiv
-
[2003]
Robust visual domain randomization for reinforcement learning
Reda Bahi Slaoui, William R Clements, Jakob N Foerster, and S´ebastien Toth. Robust visual domain randomization for reinforcement learning. arXiv preprint arXiv:1910.10537,
1910 arXiv
-
[2012]
High- performance neural networks for visual object classification
Dan C Cires ¸an, Ueli Meier, Jonathan Masci, Luca M Gambardella, and J¨urgen Schmidhuber. High- performance neural networks for visual object classification. arXiv preprint arXiv:1102.0183 ,
-
[2014]
Deepmind control suite
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690,
-
[2016]
When to use parametric models in reinforce- ment learning? arXiv preprint arXiv:1906.05243,
Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforce- ment learning? arXiv preprint arXiv:1906.05243,
1906 arXiv
-
[2017]
Parameter space noise for exploration
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv:1706.01905,
-
[2018]
Learning actionable rep- resentations from visual observations
Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, and Pierre Sermanet. Learning actionable rep- resentations from visual observations. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1577–1584. IEEE,
2018
-
[2020]
Multi-column deep neural networks for image classification
Dan Ciregan, Ueli Meier, and J ¨urgen Schmidhuber. Multi-column deep neural networks for image classification. In 2012 IEEE conference on computer vision and pattern recognition , pp. 3642–
2012
-
[2021]
Bridging the gap between value and policy based reinforcement learning
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. arXiv preprint arXiv:1702.08892,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.