Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Scaling CrossQ with Weight Normalization

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Weight normalization lets CrossQ scale to high update-to-data ratios, reaching competitive or better results than BRO on the hardest DeepMind Control tasks without needing network resets.

desk verdict CrossQ+WN's UTD-10 scaling is a real, reproducible-looking empirical result, but the paper's causal story is asserted, not measured; worth reviewing with a required diagnostic. read the letter →

arxiv 2506.03758 v1 pith:2CXMIZ2I submitted 2025-06-04 cs.LG cs.AI

Daniel Palenicek , Florian Vogt , Jan Peters This is my paper
classification cs.LGcs.AI
keywords sampleefficiencyupdate-to-dataratioweightnormalizationbatchCrossQDeepMindControlQ-biascontinuous
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CrossQ is a sample-efficient off-policy reinforcement learning algorithm built around batch normalization, but its training can be brittle and does not reliably improve when given more gradient updates per environment step. This paper identifies Q-function bias explosions and growing critic weight norms as the problems that higher update-to-data (UTD) ratios amplify. Adding weight normalization to the critic counteracts weight growth, keeps the effective learning rate constant, and stabilizes training. With weight normalization, CrossQ at UTD 10 matches or beats BRO on the hardest DeepMind Control dog and humanoid tasks, and its sample efficiency improves monotonically as UTD increases from 1 to 10. The practical payoff is that high-UTD training no longer needs periodic network resets.

What carries the argument

Weight normalization is the central mechanism: each critic weight vector is written as a learnable direction $\mathbf{v}$ scaled by a learnable norm magnitude $g$, so a layer computes $W x = g (\mathbf{v}/\lVert\mathbf{v}\rVert) x$. Because CrossQ's batch normalization already makes the network scale-invariant in its weights, the norm part of weight normalization can counteract the slow growth of critic weights, and the gradient scaling that normalization induces keeps the effective learning rate constant. This is what stabilizes Q-bias and lets the algorithm scale to UTD 10.

What would settle it

Run CrossQ with weight normalization at UTD 10 while logging critic weight norms and Q-bias, and compare against CrossQ with tuned weight decay that holds the norms to the same range; if the weight-decay version scales just as well, then weight normalization's scale-invariant constant-effective-learning-rate property is not the operative mechanism.

Watch

Extended reading notes

Core claim

The paper claims that CrossQ with weight normalization reliably scales to higher update-to-data ratios across 15 DeepMind Control tasks, reaching performance competitive with or better than BRO on the complex dog and humanoid environments while remaining reset-free. The authors argue that vanilla CrossQ's instability at high UTD ratios is driven by Q-bias explosion and unbounded growth of critic weight norms; weight normalization, which reparameterizes each weight as a direction times a scalar magnitude, prevents the weight norms from growing, preserves the scale-invariance that batch normalization gives the critic, and keeps the effective learning rate constant. As a result, training curves are stable, sample efficiency improves as UTD goes from 1 to 10, and the method beats a SAC baseline that uses UTD 32 with periodic resets while avoiding resets' performance drops.

Load-bearing premise

The argument assumes that high-UTD instability in CrossQ is caused by Q-bias explosion and growing critic weight norms, and that weight normalization fixes those root causes; if the true driver is distribution shift, batch-normalization running statistics, or optimizer interactions, weight normalization's benefit would be incidental and the scaling may not generalize beyond the tested tasks and UTD range.

Editorial extensions

If this is right

  • At UTD 10, CrossQ with weight normalization is competitive with or better than BRO on the dog and humanoid DeepMind Control tasks, without the periodic resets that BRO-style pipelines use.
  • Increasing UTD from 1 to 10 produces ordered, reliable gains in IQM return across 15 tasks, so users can trade more compute for fewer environment interactions.
  • The method beats SAC with UTD 32 plus resets on the aggregated benchmark, suggesting that reset-free training need not sacrifice sample efficiency.
  • Vanilla CrossQ needs per-environment tuning on DMC, and adding weight normalization removes much of that tuning burden.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is that weight normalization should also stabilize other batch-normalized off-policy algorithms, not just CrossQ, when the UTD ratio is raised.
  • Because the method avoids resets, it is a better fit for continual or real-world deployment where catastrophic performance drops mid-training are costly; this is an editorial inference, not tested in the paper.
  • The paper leaves sparse-reward and one specific locomotion task (pendulum-swingup, hopper-hop) behind, so testing weight normalization combined with reward shaping or auxiliary exploration on those tasks could reveal whether the failure is environmental or fundamental.
  • A direct head-to-head with weight decay or spectral normalization at matched weight norms would test whether weight normalization's constant-effective-learning-rate property, rather than plain norm control, is what causes the scaling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper studies scaling CrossQ, a sample-efficient model-free RL algorithm that uses batch normalization, to higher update-to-data (UTD) ratios. The authors report that vanilla CrossQ is brittle on DeepMind Control and that high UTD ratios lead to Q-bias explosion and growing critic weight norms. They propose adding weight normalization (WN) to CrossQ, arguing that the scale invariance and constant effective learning rate counteract these effects. They present aggregate IQM learning curves over 15 DMC tasks showing reliable scaling for UTD ratios in {1,2,5,10}, and compare CrossQ+WN at UTD=10 against BRO and SAC with UTD=32 and resets, reporting competitive performance on dog and humanoid tasks without resets.

Significance. The proposed modification is simple and could be practically valuable if the scaling result holds, since it avoids network resets and preserves CrossQ's computational efficiency. The empirical evaluation uses a reasonable protocol: 15 DMC environments, 10 seeds, and IQM aggregation with confidence intervals. However, the causal mechanism is asserted rather than demonstrated, and the main baseline comparison relies on externally published BRO curves, so the significance is contingent on additional evidence. The paper also omits the experimental details needed for reproduction.

major comments (4)
  1. [Section 4, Figures 2 and 3, Abstract] The causal story that WN prevents Q-bias explosion and critic weight growth via scale invariance and a constant effective learning rate is not supported by any diagnostic for CrossQ+WN. Figure 1 reports Q-bias and parameter norms only for SAC, CrossQ, and CrossQ at UTD 32; the paper never reports these quantities for the proposed method. The abstract and Section 4 present this mechanism as the explanation for the scaling gains, so the paper should add Q-bias and weight-norm curves for CrossQ+WN on the same environments, and ideally a plasticity measure, to show that WN actually changes the hypothesized quantities. Without that, the empirical gains may be due to initialization geometry, implicit regularization, or optimizer interactions, and the claim that WN prevents loss of plasticity is unsubstantiated.
  2. [Section 4, Figure 3] The BRO baseline is not run in the same harness; the paper states, 'for the BRO baseline we plot the official training data the authors published.' This introduces uncontrolled differences in environment versions, reward wrappers, evaluation protocol, hyperparameter settings, and seeding. This prevents a controlled comparison and directly affects the abstract's claim of being competitive or superior to BRO. The authors should either re-run BRO in the same harness with the same seeds and evaluation cadence, or clearly label the comparison as indicative and soften the superiority claims unless differences are shown to be robust.
  3. [Experimental setup (throughout)] The paper does not provide an experimental setup section or reproducibility details: the placement of weight normalization (which layers, actor and/or critic), gain initialization, optimizer hyperparameters, network widths, batch size, replay buffer size, learning rate schedule, environment action-repeat, and evaluation protocol are not stated. Consequently the empirical claims are not reproducible and the UTD scaling comparison cannot be independently checked. Please add a complete experimental details section or appendix.
  4. [Section 4, Figure 2] The claim that sample efficiency scales reliably with increasing UTD ratios is supported only by aggregate IQM curves. Per-environment final performance and confidence intervals are not shown, and no statistical test of monotonic improvement across UTD ratios is provided. The text says that individual learning curves are 'nicely ordered' but omits them due to space constraints. Please include per-task final-return tables or individual curves in a supplementary appendix and report the probability of improvement across tasks, so the reader can confirm that the scaling is consistent rather than driven by a few environments.
minor comments (4)
  1. [Throughout] There are several typographical errors, including 'THe' in Section 4, 'achieces' in the Figure 3 caption, 'environemnts' near the Figure 3 discussion, and 'network parameterized' in Section 2. A careful proofread is needed.
  2. [Figures 2 and 3 captions] Figure 2 reports 95% confidence intervals while Figure 3 uses 90% inter-percentile ranges; the captions should state why different summary intervals are used and define the aggregation method consistently.
  3. [Section 3, Figure 1] The quantity 'Q-bias' is not defined in the text or caption. Please define it precisely, e.g., the average difference between the target Q-value and the predicted Q-value under the current policy, and specify over which states and actions it is computed.
  4. [Section 4] The term 'effective learning rate' is cited to Van Laarhoven (2017) but not defined in the paper. A one-sentence definition would help readers who are not familiar with that line of work.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scaling claim is evaluated against external benchmark tasks and no fitted quantity is renamed as a prediction.

full rationale

The paper's central claim—that adding weight normalization to CrossQ enables stable scaling to higher update-to-data ratios—is tested against external DeepMind Control benchmarks and compared with BRO and SAC with resets. No fitted parameter is renamed as a prediction, and no equation defines the outcome in terms of the input. Figure 1 documents Q-bias and weight-norm growth for vanilla CrossQ, but the paper does not directly measure these quantities for CrossQ+WN; this is a missing mechanistic verification, not circularity, because the empirical scaling result does not reduce to that diagnostic. The only author-overlapping citation is Bhatt et al. (2024) for CrossQ itself, which is prior published work used as the base algorithm rather than an argument that forces the present result. The citations to Van Laarhoven (2017) and Lyle et al. (2024) supply external mathematical and empirical support for the scale-invariance and effective-learning-rate mechanism. No self-definitional, fitted-input-called-prediction, uniqueness-imported, ansatz-smuggled, or renaming pattern is present. The result is therefore not circular; at most it carries an unverified causal hypothesis about why weight normalization helps.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper does not fit constants to a held-out test set. The ledger records the chosen operating point UTD=10, unstated normalization placement and inherited CrossQ hyperparameters, and four domain assumptions connecting the empirical results to the stated mechanism.

free parameters (3)
  • Update-to-data ratio (UTD) = 10 for headline runs; 1, 2, 5 also tested
    The central comparison uses UTD=10, an operating point chosen by the authors. The ordering across UTD values supports scaling, but the headline number is a design choice rather than a fitted constant.
  • Weight normalization placement and gain initialization = not stated in paper
    The paper does not specify which layers receive weight normalization, how the gain is initialized, or how it interacts with batch normalization. These choices can affect the empirical outcome.
  • CrossQ base hyperparameters = not stated in paper
    Learning rate, batch size, network widths, and batch normalization placement are inherited from CrossQ and not reported here, which matters for reproducing the exact scaling behavior.
assumptions (4)
  • domain assumption CrossQ's batch-normalized critic is scale invariant, so weight normalization changes weight magnitudes without changing the computed function.
    Invoked in Section 4 to justify weight normalization. It relies on Van Laarhoven 2017 and on batch normalization behaving as assumed inside CrossQ, including the target-distribution handling introduced by Bhatt et al. 2024.
  • domain assumption Growing critic weight norms and Q-bias explosion cause the instability observed at high update-to-data ratios, not merely correlate with it.
    Section 3 links poor CrossQ performance to fluctuating Q-bias and large weight norms, but no intervention or ablation isolates causation.
  • domain assumption Keeping the effective learning rate constant via weight normalization prevents loss of plasticity, transferring Lyle et al.'s layer-normalization plus weight-normalization result to CrossQ's batch-normalization setting.
    Section 4 borrows the effective-learning-rate argument from prior discrete-action reinforcement learning; no plasticity measurement is made in this paper.
  • domain assumption BRO's published official training curves are directly comparable to the authors' own runs in this paper.
    Figure 3 uses official BRO data for computational reasons, but environment versions, seeds, and evaluation protocol are not confirmed to match the authors' runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling CrossQ with Weight Normalization." pith.science (2026). https://pith.science/paper/2CXMIZ2I

@misc{pith2026250603758,
  author       = {Pith},
  title        = {Pith review of: Scaling CrossQ with Weight Normalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2CXMIZ2I}},
  note         = {Machine review of arXiv:2506.03758}
}
read the original abstract

Reinforcement learning has achieved significant milestones, but sample efficiency remains a bottleneck for real-world applications. Recently, CrossQ has demonstrated state-of-the-art sample efficiency with a low update-to-data (UTD) ratio of 1. In this work, we explore CrossQ's scaling behavior with higher UTD ratios. We identify challenges in the training dynamics which are emphasized by higher UTDs, particularly Q-bias explosion and the growing magnitude of critic network weights. To address this, we integrate weight normalization into the CrossQ framework, a solution that stabilizes training, prevents potential loss of plasticity and keeps the effective learning rate constant. Our proposed approach reliably scales with increasing UTD ratios, achieving competitive or superior performance across a range of challenging tasks on the DeepMind control benchmark, notably the complex dog and humanoid environments. This work eliminates the need for drastic interventions, such as network resets, and offers a robust pathway for improving sample efficiency and scalability in model-free reinforcement learning.

Figures

Figures reproduced from arXiv: 2506.03758 by the authors.

Figure 1
Figure 1. Q-bias and weight norms. In general CrossQ shows much larger weight norms than SAC. High UTD can emphasize this problem even more. Large and high variance Q-bias can often be linked to poor performance. to repeatedly start learning from scratch, it is also not very compute efficient. Finally, the exact reasons why parameter resets work so well in practice are not well understood [Li et al., 2023]. Instead of resetti… view at source ↗
Figure 3
Figure 3. CrossQ WN UTD=10 against baselines. We compare our proposed CrossQ WN UTD=10 against two baselines, BRO [Nauman et al., 2024] and SAC with UTD=32 and resets. Results are reported on all 15 DMC tasks and the IQMand 90% inter percentile ranges over 10 random seeds. The leftmost plots (black border) provide aggregated results over the dog and humanoid environments and over 15 environments, respectively. Our proposed ap… view at source ↗
Figure 2
Figure 2. CrossQ WN UTD scal￾ing behavior. We plot the IQM return and 95% confidence inter￾vals for different UTD ratios ∈ {1, 2, 5, 10}. The results are ag￾gregated over 15 DMC environ￾ments and 10 random seeds each according to Agarwal et al. [2021]. The sample efficiency scales reli￾ably with increasing UTD ratios. Weight normalization allows CrossQ to scale effectively. Next, we provide em￾pirical evidence for our hypothe… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Replacing the hard target-copy in DQN with a Q-value-sensitivity-weighted merge of the last K network copies yields competitive Atari performance, but only marginally beats architecture-matched baselines.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Agarwal, M

    R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 2021

    work page 2021

  2. [2]

    Bhatt, D

    A. Bhatt, D. Palenicek, B. Belousov, M. Argus, A. Amiranashvili, T. Brox, and J. Peters. Cross Q : Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In International conference on learning representations, 2024

    work page 2024

  3. [3]

    X. Chen, C. Wang, Z. Zhou, and K. Ross. Randomized ensembled double Q -learning: Learning fast without a model. In International conference on learning representations, 2021

    work page 2021

  4. [4]

    A. I. Cowen-Rivers, D. Palenicek, V. Moens, M. A. Abdullah, A. Sootla, J. Wang, and H. Bou-Ammar. Samba: Safe model-based & active reinforcement learning. Machine learning, 2022

    work page 2022

  5. [5]

    D'Oro, M

    P. D'Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In International conference on learning representations, 2022

    work page 2022

  6. [6]

    Hiraoka, T

    T. Hiraoka, T. Imagawa, T. Hashimoto, T. Onishi, and Y. Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. In International conference on learning representations, 2021

    work page 2021

  7. [7]

    Huang, J

    L. Huang, J. Qin, Y. Zhou, F. Zhu, L. Liu, and L. Shao. Normalization techniques in training dnns: Methodology, analysis and application. IEEE transactions on pattern analysis and machine intelligence, 2023

    work page 2023

  8. [8]

    Hussing, C

    M. Hussing, C. Voelcker, I. Gilitschenski, A.-m. Farahmand, and E. Eaton. Dissecting deep rl with high update ratios: Combatting value overestimation and divergence. arXiv preprint arXiv:2403.05996, 2024

    arXiv 2024

Show all 18 references
  1. [9]

    Janner, J

    M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In Advances in neural information processing systems, 2019

    2019

  2. [10]

    Q. Li, A. Kumar, I. Kostrikov, and S. Levine. Efficient deep reinforcement learning requires regulating overfitting. In International conference on learning representations, 2023

    2023

  3. [11]

    C. Lyle, Z. Zheng, K. Khetarpal, J. Martens, H. van Hasselt, R. Pascanu, and W. Dabney. Normalization and effective learning rates in reinforcement learning. In Neural information processing systems, 2024

    2024

  4. [12]

    Nauman, M

    M. Nauman, M. Ostaszewski, K. Jankowski, P. Miłoś, and M. Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Advances in neural information processing systems, 2024

    2024

  5. [13]

    Nikishin, M

    E. Nikishin, M. Schwarzer, P. D’Oro, P.-L. Bacon, and A. Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, 2022

    2022

  6. [14]

    M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  7. [15]

    R. S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Machine learning, 1990

    1990

  8. [16]

    R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018

    2018

  9. [17]

    Todorov, T

    E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In International conference on intelligent robots and systems, 2012

    2012

  10. [18]

    Van Laarhoven

    T. Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350, 2017

    2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.