Pith. sign in

REVIEW 3 major objections 5 minor 59 references

Optimistic critics can empower small actors

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that shrinking an actor in SAC or DrQ degrades performance mostly because the minimum-of-two-critics rule makes the critic underestimate values, and that replacing that min with the mean or max of the two critics largely…

desk verdict A careful empirical study on small actors that overclaims the data-collection mechanism; the mean/max rescue is real and the causal story is underdetermined. read the letter →

arxiv 2506.01016 v3 pith:UQOYGOA2 submitted 2025-06-01 cs.LG stat.ML

Olya Mastikhina , Dhruv Sreenivas , Pablo Samuel Castro This is my paper
classification cs.LGstat.ML
keywords actor-criticsoftsmallactorsasymmetricvalueunderestimationoptimisticcriticscontinuouscontrolreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the severe performance drop from drastically shrinking the actor in Soft Actor-Critic and DrQ is not a hard limit of the actor's reduced capacity but rather a side effect of value underestimation in the critics. When the actor shrinks, the standard minimum-of-two-critics rule produces overly conservative Q-estimates, the actor under-explores, and the data it collects becomes poor, causing the critics to overfit. The central intervention is simple: replace the min with the mean or the maximum of the two critic Q-estimates in both the critic and actor losses, which raises value estimates, restores policy entropy, and makes small actors competitive with regular-sized actors. If this holds, it gives a cheap, practical recipe for training small, deployable actors in resource-constrained settings without giving up performance.

What carries the argument

The central object is the aggregation operator over the two critic heads in the SAC/DrQ losses, equations (4) and (5): the standard min, versus the proposed mean or max. This operator sets the target values and the actor-gradient signal; the paper shows that varying only this operator, while keeping network sizes fixed, substantially alters value estimates and downstream data quality.

What would settle it

Train the smallest actor under the standard min-rule critic on a fixed dataset collected by a full-size actor, as in the paper's bottom row of figure 2. If this data-sufficient small actor does not recover most of the performance gap, the claim that poor data collection is the main channel would be contradicted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the min-operation used by SAC and DrQ to combine two critic heads is the key bottleneck when the actor is small: it under-estimates Q-values, the actor under-explores as a result, and the data it collects is poor, which in turn makes the critics overfit. Replacing min with the mean or the maximum of the two Q-estimates in both the critic and actor losses raises validation Q-values, restores entropy, and boosts final performance of actors with only 1% of the default parameters; in some environments the small actor with mean/max critics even beats the full actor trained with the min rule.

Load-bearing premise

The paper assumes that the performance drop from small actors is caused by value underestimation leading to poor data collection, rather than by the small actor's reduced representational capacity, since the evidence is correlational and the main intervention changes the actor's learning objective as well as the data it collects.

Editorial extensions

If this is right

  • Small actors (down to 1% of baseline parameters) can be trained with a one-line modification, mean or max critics, to reach near or even above regular-actor performance.
  • Asymmetric actor-critic with a small, deployable actor becomes more practical, preserving inference-cost and interpretability benefits without a large performance penalty.
  • The mean/max fix also helps when the actor receives limited inputs (partial observability), suggesting that addressing underestimation is a general strategy for constrained actors.
  • The findings question the automatic use of the min rule in conservative actor-critic methods, motivating more nuanced bias correction rather than blind overestimation avoidance.
  • Critic regularization alone does not recover the performance, highlighting that the value-bias channel, not general overfitting, is what matters most for small actors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's causal story, that poor data collection via underestimation is the main channel, is supported by correlations and interventions but not fully isolated; a direct test would train a small actor with the min-critic on data from a regular actor and compare recovery, which the paper only partially does.
  • The finding likely extends to other min-based actor-critic algorithms such as TD3, though those are not tested here; the same swap to mean or max could relieve their underestimation bias in constrained settings.
  • In sample-scarce or expensive environments, using optimistic critics may be a more effective remedy for small actors than adding regularization or resets, since it directly targets the value bias rather than its consequences.
  • A testable extension would sweep between min and max (e.g., a weighted average of the two critics) to characterize the bias-variance trade-off and find the optimal optimism level for a given actor size.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper empirically studies what happens when the actor network in Soft Actor-Critic (SAC, state-based) and DrQ (pixel-based) is made much smaller than the critic on DeepMind Control tasks. It documents that smaller actors degrade final performance, are associated with higher critic overfitting (o_phi, dormant neurons), lower relative Q-values, and lower policy entropy. Data-quality experiments show that a regular-sized actor trained on data collected by a smaller actor loses performance, and that in some SAC environments a small actor improves when trained on data from a regular actor, while DrQ shows no clear transfer benefit. The central intervention replaces the minimum over the two critics with their mean or maximum in both the critic target and the actor objective; this substantially improves small-actor performance in SAC and, to a lesser extent, in DrQ, whereas critic regularization and resets do not. The paper concludes that poor data collection due to value underestimation is a main cause of small-actor degradation and that alleviating underestimation can empower small actors.

Significance. The main positive result---that mean/max critics can markedly improve the performance of small actors---is practically relevant for deployment-oriented actor-critic methods, and the paper presents it with a broad set of environments, 10--20 seeds, and confidence intervals. The authors also report null or weak results for DrQ and for critic-regularization baselines, which is good empirical practice. The paper ships no code, but the experimental setup is described in enough detail to be reproducible from existing SAC/DrQ implementations. If the causal story were fully established, the work would also contribute to the understanding of value bias in actor-critic methods. However, as it stands, the evidence for the 'value underestimation leads to poor data collection' mechanism is not isolated, and one piece of evidence used for underestimation is confounded. These issues do not invalidate the empirical intervention, but they do affect the strength of the explanatory claims.

major comments (3)
  1. [Section 3.2, Fig. 3] The claim that smaller actors 'result in Q-value underestimation' is not supported by the presented evidence. The figure plots average critic Q-values relative to the regular actor, but a smaller actor with lower true returns will also have lower unbiased Q-estimates; the relative comparison therefore does not demonstrate underestimation. To support this mechanistic claim, the authors should compare Q-estimates to empirical (Monte Carlo) returns or examine the sign and magnitude of TD residuals. Without such a comparison, the term 'underestimation' is an interpretation rather than a measurement, and it is load-bearing for the paper's causal narrative.
  2. [Section 4.1, Eqs. (4)-(5)] The mean/max intervention replaces the min operator in both the critic target (Eq. 4) and the actor objective (Eq. 5). This changes the actor's policy-gradient signal directly, by removing the min-clipping bias in action selection and by altering the coupling with the entropy bonus, so the observed improvement cannot be attributed solely to improved data collection. The data-quality experiments in Section 3.1 (Fig. 2 and Fig. 9) are correlational and partial: they do not show that a small actor trained on high-quality data from a regular actor fully recovers performance, and they do not measure state coverage. To make the 'mostly due to poor data collection' claim, please add an experiment that varies data collection while keeping the min rule (for example, training the xs actor offline on a buffer collected by the r actor), or rephrase the conclusions to distinguish the empirical intervention from the causal mechanism.
  3. [Abstract and Section 4 (opening paragraph)] The causal attribution is stated more strongly in the abstract ('poor data collection, due to value underestimation, as one of the main causes') and in the opening of Section 4 ('largely due to poor data collection') than the evidence supports, given the issues in the previous two comments. If the isolating experiment is not added, the authors should soften these statements to say that the results are consistent with this mechanism rather than that the mechanism is established.
minor comments (5)
  1. [Fig. 4 caption] The caption says 'The top row displays final performance while the bottom two rows display performance throughout training,' but the figure has three rows and the text refers to top/middle/bottom rows with different groupings for SAC and DrQ; please clarify which row corresponds to which setting.
  2. [Section 2, Eq. (1)] The TD error in Eq. (1) is defined with an absolute value, but later the notation T D_phi is used in the definition of o_phi without a formal definition; please define T D_phi explicitly to avoid ambiguity.
  3. [Section 3, actor size labels] The labels r/m/s/xs are introduced only in Section 3, but they would be helpful in Section 2.1 where the architectures are described; consider defining them earlier or adding a pointer.
  4. [Section 4.2, L2 distance from initialization] The L2 distance from initialization coefficient is tuned on a single environment (quadruped-run) and then used elsewhere; please state the chosen value and clarify that this tuning is an exception to the otherwise default hyperparameters, as this affects the interpretation of the regularization results.
  5. [Fig. 9 caption] The figure labels 'online' and 'offline' are not defined in the caption; please clarify whether 'offline' refers to training on a fixed buffer collected by the regular actor and 'online' to the standard interactive training.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claims rest on external performance benchmarks and interventions; only minor non-load-bearing self-citations and a small hyperparameter-tuning choice are present.

full rationale

The paper's central empirical claims—smaller actors degrade performance, critics overfit more, and mean/max critic aggregation improves returns—are evaluated against external DeepMind Control suite returns, not derived from fitted quantities. Section 3.1's data-quality experiment (training a regular-sized actor on data collected by smaller actors) and Section 4.1's mean/max replacement in Eqs. (4)–(5) are interventions whose outcomes are independently measured; neither reduces to a definition. The Q-underestimation and entropy correlations in Section 3.2 are evidence for a hypothesized mechanism, and the causal attribution to 'poor data collection' is underdetermined because the mean/max intervention also changes the actor's policy-gradient objective directly (Eq. 5). That is a threat to internal validity, not circularity: the performance outcome is not defined in terms of the intervention. Self-citations (Sokar et al. 2023 for dormant neurons, Ostrovski et al. 2021 for the tandem effect, Ceron et al. 2024 for hyperparameter sensitivity) are used as metric or hypothesis citations and are not load-bearing for the main result. The L2-from-init coefficient in Section 4.2 is tuned on quadruped-run, but the paper does not present L2i as a central prediction and reports that regularization does not rescue performance; this is a minor selection-on-validation issue, not a circular derivation. Overall, the paper is self-contained against external benchmarks, so a score of 2 reflects only minor non-load-bearing self-citation.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. Its central claim depends on standard RL assumptions, the validity of the chosen benchmark suite, the representativeness of a fixed validation buffer, and the reliability of proxy metrics for overfitting and plasticity. The only explicitly fitted parameter is the L2-from-init coefficient, tuned on one environment.

free parameters (1)
  • L2 distance from initialization coefficient (lambda) = 1e-7
    Tuned on quadruped-run over the range [5e-8, 1e-4] in increments of 0.5, then applied to all environments. This is the only explicitly tuned hyperparameter claimed to affect the regularization intervention.
assumptions (4)
  • standard math Standard MDP formalization and TD learning framework.
    The paper relies on the standard MDP definitions and the TD error objective in Section 2 without re-deriving them.
  • domain assumption DMC with default hyperparameters is a valid testbed for actor-critic methods.
    The paper uses the default SAC/DrQ settings from Haarnoja et al. and Yarats et al. and treats DMC tasks as canonical continuous-control benchmarks. It acknowledges hyperparameter sensitivity as a limitation.
  • domain assumption The validation buffer from a regular SAC run with a different seed is representative for measuring critic overfitting (o_phi) and Q-value levels.
    o_phi and validation Q-values are computed on a fixed set of 11,000 transitions collected by an unmodified SAC agent. If this buffer is not representative of the small actor's state distribution, the measured underestimation and overfitting could be artifacts. This enters in Section 3.
  • domain assumption o_phi, dormant neuron fraction, effective rank, and parameter norm are valid proxies for overfitting, plasticity loss, and optimization difficulty.
    These metrics are taken from Nauman et al. 2024, Sokar et al. 2023, Kumar et al. 2021, and the authors list them as indicative. They are used to support the narrative but are not independently validated in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimistic critics can empower small actors." pith.science (2026). https://pith.science/paper/UQOYGOA2

@misc{pith2026250601016,
  author       = {Pith},
  title        = {Pith review of: Optimistic critics can empower small actors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQOYGOA2}},
  note         = {Machine review of arXiv:2506.01016}
}
read the original abstract

Actor-critic methods have been central to many of the recent advances in deep reinforcement learning. The most common approach is to use symmetric architectures, whereby both actor and critic have the same network topology and number of parameters. However, recent works have argued for the advantages of asymmetric setups, specifically with the use of smaller actors. We perform broad empirical investigations and analyses to better understand the implications of this and find that, in general, smaller actors result in performance degradation and overfit critics. Our analyses suggest poor data collection, due to value underestimation, as one of the main causes for this behavior, and further highlight the crucial role the critic can play in alleviating this pathology. We explore techniques to mitigate the observed value underestimation, which enables further research in asymmetric actor-critic methods.

Figures

Figures reproduced from arXiv: 2506.01016 by the authors.

Figure 1
Figure 1. Decreasing the size of the actor in SAC decreases performance (top row) and in￾creases overfitting in the critics, as measured by oϕ (Nauman et al., 2024, middle row) and dor￾mant neurons (Sokar et al., 2023, bottom row). In the top row, the y-axis is kept fixed to show the relative performance impacts across environments; this becomes impractical for the metrics in the middle and bottom row. We report the final per… view at source ↗
Figure 2
Figure 2. Evaluating the impact of data quality collecting by actors of varying sizes. Top row: the black bars denote training regularly-sized πr on data collected by one of the smaller actors, while the colored bars indicate each actor trained on its own data. Bottom row: the smallest (xs) actor trained on data from the largest (r) actor, for varying fractions of training length. In both rows we report the final performance,… view at source ↗
Figure 3
Figure 3. Decreasing the size of the actor results in Q-value underestimation and reduced policy entropy. In the top row we estimate the average Q-values on a batch of data gathered during evaluation, and plot the values relative to the baseline r. In the bottom row we compute the entropy of the policy π and plot the values relative to the entropy of the baseline r. In both cases we report the values obtained at the end of tr… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Taking the mean or max of the two critics can empower smaller actors in SAC (top and middle rows) and on the smallest actor in DrQ (bottom row). Replacing the minimums in equations 4 and 5 with mean and max can help reduce Q-value underestimation and boost performance.…
Figure 5
Figure 5. Figure 5: Impact of attempted modifications on the final performance (top), mean entropy of the actor’s action distribution (middle), and validation Q-value estimation (bottom) of the smallest actor in SAC. The values are relative to the smallest actor in unmodified SAC. Bars in…
Figure 6
Figure 6. Figure 6: Taking the mean of the two critics can help deal with partial observability in the actor. We zero-out 2/3 of the inputs into the actor and compare the performance when using the min or mean of the two critics. It is often necessary to provide the actor with less inform…
Figure 7
Figure 7. Figure 7: Decreasing the size of the actor in DrQ decreases performance (top row) and in￾creases overfitting in the critic, as measured by oϕ (Nauman et al., 2024, bottom row). The solid lines represent mean performance, while the shaded area represents the 95% confidence interv…
Figure 8
Figure 8. Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Training the smallest actor on data collected by the largest, high-performing actor does not appear to lead to a clear improvement in performance in DrQ across a suite of en￾vironments, possibly due to high variance, although an improvement trend might be suggested for…
Figure 10
Figure 10. Figure 10: Impact of various modifications on final performance and validation Q-values in DrQ. In the resetting experiments, e reset every 100, 000 steps, and only reset the critics. As in (Nikishin et al., 2022), we only reset the MLP of the critics, and leave the encoder unto…
Figure 11
Figure 11. Figure 11: Decreasing the size of the actor results in Q-value underestimation and reduced policy entropy. In the top row we estimate the average Q-values on a batch of data gathered during evaluation, and plot the values relative to the baseline r. In the bottom row we compute …
Figure 12
Figure 12. Figure 12: Impact of critic regularizations on downstream performance with actors of varying sizes. Each table row corresponds to one of the normalization mechanisms explored, each column indicates the actor size used, and the value in each cell denotes the change relative to th…
Figure 13
Figure 13. Figure 13: The impact of small actors on a number of metrics related to plasticity as measured on the critics. We also evaluate these metrics when using the mean and max of the two critics, as discussed in section 4.1 [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: The impact of small actors on a number of metrics related to plasticity as measured on the actor. We also evaluate these metrics when using the mean and max of the two critics, as discussed in section 4.1. We define oϕ on the actor as oϕ := EDH EDV H , where H is the …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 39 canonical work pages

  1. [1]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450

    arXiv 2016

  2. [2]

    Unbiased Asymmetric Reinforcement Learning under Partial Observability

    Andrea Baisero and Christopher Amato. Unbiased Asymmetric Reinforcement Learning under Partial Observability . In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , AAMAS '22, pp.\ 44--52, Richland, SC, May 2022. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978-1-4503-9213-6

    work page 2022

  3. [3]

    On the consistency of hyper-parameter selection in value-based deep reinforcement learning

    Johan Samir Obando Ceron, Jo \ a o Guilherme Madeira Ara \'u jo, Aaron Courville, and Pablo Samuel Castro. On the consistency of hyper-parameter selection in value-based deep reinforcement learning. In Reinforcement Learning Conference, 2024. URL https://openreview.net/forum?id=szUyvvwoZB

    work page 2024

  4. [4]

    Better exploration with optimistic actor critic

    Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/...

    work page 2019

  5. [5]

    Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan D. Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean - Marc Moret, Seb Noury, Federico Pesamosca, David Pfau, Ol...

    2022

  6. [6]

    IMPALA : Scalable Distributed Deep - RL with Importance Weighted Actor - Learner Architectures

    Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA : Scalable Distributed Deep - RL with Importance Weighted Actor - Learner Architectures . In Proceedings of the 35th International Conference on Machine Learning . PMLR, July 2018

    work page 2018

  7. [7]

    On Interpretability of Artificial Neural Networks : A Survey

    Feng-Lei Fan, Jinjun Xiong, Mengzhou Li, and Ge Wang. On Interpretability of Artificial Neural Networks : A Survey . IEEE Transactions on Radiation and Plasma Medical Sciences, 5 0 (6), November 2021. ISSN 2469-7303

    work page 2021

  8. [8]

    Foundation models in robotics: Applications , challenges, and the future

    Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, Brian Ichter, Danny Driess, Jiajun Wu, Cewu Lu, and Mac Schwager. Foundation models in robotics: Applications , challenges, and the future. The International Journal of Robotics Research, September 2024. ISSN 0278-3649

    work page 2024

Show all 59 references
  1. [9]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Swe...

    2018

  2. [10]

    Soft Actor - Critic : Off - Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor - Critic : Off - Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor . In Proceedings of the 35th International Conference on Machine Learning . PMLR, July 2018 a

    2018

  3. [11]

    Soft actor-critic algorithms and applications

    Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/1812.05905, 2018 b . URL http://arxiv.org/abs/1812.05905

    2018 arXiv

  4. [12]

    Array programming with numpy

    Charles R Harris, K Jarrod Millman, St \'e fan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585 0 (7825): 0 357--362, 2020

    2020

  5. [13]

    Double q-learning

    Hado Hasselt. Double q-learning. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010

    2010

  6. [14]

    Deep reinforcement learning with double q-learning

    Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pp.\ 2094–2100. AAAI Press, 2016

    2016

  7. [15]

    Wd3: Taming the estimation bias in deep reinforcement learning

    Qiang He and Xinwen Hou. Wd3: Taming the estimation bias in deep reinforcement learning. In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), pp.\ 391–398. IEEE, November 2020. doi:10.1109/ictai50040.2020.00068. URL http://dx.doi.org/10.110...

    2020

  8. [16]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015. URL https://api.semanticscholar.org/CorpusID:7200347

    2015 arXiv

  9. [17]

    Toward General - Purpose Robots via Foundation Models : A Survey and Meta - Analysis , October 2024

    Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Tianyi Zhang, Hao-Shu Fang, Shibo Zhao, Shayegan Omidshafiei, Dong-Ki Kim, Ali-akbar Agha-mohammadi, Katia Sycara, Matthew Johnson-Roberson, Dhruv Batra, Xiaolong Wang, ...

    2024

  10. [18]

    Matplotlib: A 2d graphics environment

    John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9 0 (03): 0 90--95, 2007

    2007

  11. [19]

    Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning

    Haque Ishfaq, Guangyuan Wang, Sami Nur Islam, and Doina Precup. Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=FvQsk3la17

    2025

  12. [20]

    Littman, and Anthony R

    Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artif. Intell., 101 0 (1–2): 0 99–134, May 1998. ISSN 0004-3702

    1998

  13. [21]

    Plasticity loss in deep reinforcement learning: A survey, 2024

    Timo Klein, Lukas Miklautz, Kevin Sidak, Claudia Plant, and Sebastian Tschiatschek. Plasticity loss in deep reinforcement learning: A survey, 2024. URL https://arxiv.org/abs/2411.04832

    2024 arXiv

  14. [22]

    Jupyter Notebooks a publishing format for reproducible computational workflows

    Thomas Kluyver , Benjain Ragan-Kelley , Fernando P \'e rez , Brian Granger , Matthias Bussonnier , Jonathan Frederic , Kyle Kelley , Jessica Hamrick , Jason Grout , Sylvain Corlay , Paul Ivanov , Dami \'a n Avila , Safia Abdalla , Carol Willing , and Jupyter Development Team ....

    2016 doi

  15. [23]

    JAXRL: Implementations of Reinforcement Learning algorithms in JAX , 10 2021

    Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX , 10 2021. URL https://github.com/ikostrikov/jaxrl

    2021

  16. [24]

    Implicit under-parameterization inhibits data-efficient deep reinforcement learning

    Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=O9bnihsFfXU

    2021

  17. [25]

    Maintaining plasticity in continual learning via regenerative regularization, 2024

    Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization, 2024. URL https://openreview.net/forum?id=lyoOWX0e0O

    2024

  18. [26]

    Efficient deep reinforcement learning requires regulating overfitting, 2023 a

    Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting, 2023 a . URL https://arxiv.org/abs/2304.10466

    2023 arXiv

  19. [27]

    Realistic actor-critic: A framework for balance between value overestimation and underestimation

    Sicen Li, Qinyun Tang, Yiming Pang, Xinmeng Ma, and Gang Wang. Realistic actor-critic: A framework for balance between value overestimation and underestimation. Frontiers in Neurorobotics, 16, 2023 b . ISSN 1662-5218. doi:10.3389/fnbot.2022.1081242. URL https://www.frontiersin...

    2023

  20. [28]

    Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond

    Xuhong Li, Haoyi Xiong, Xingjian Li, Xuanyu Wu, Xiao Zhang, Ji Liu, Jiang Bian, and Dejing Dou. Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond. Knowledge and Information Systems, 64 0 (12): 0 3197--3234, December 2022. ISSN 0219-3116

    2022

  21. [29]

    Lillicrap, Jonathan J

    Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representatio...

    2016 arXiv

  22. [30]

    Visual whole-body control for legged loco-manipulation

    Minghuan Liu, Zixuan Chen, Xuxin Cheng, Yandong Ji, Ri-Zhao Qiu, Ruihan Yang, and Xiaolong Wang. Visual whole-body control for legged loco-manipulation. In 8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id=cT2N3p1AcE

    2024

  23. [31]

    Regularization matters in policy optimization - an empirical study on continuous control

    Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell. Regularization matters in policy optimization - an empirical study on continuous control. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=yr1mzrH3IC

    2021

  24. [32]

    Clare Lyle, Zeyu Zheng, Khimya Khetarpal, H. V. Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. ArXiv, abs/2402.18762, 2024. URL https://api.semanticscholar.org/CorpusID:268063557

    2024 arXiv

  25. [33]

    A deeper understanding of state-based critics in multi-agent reinforcement learning

    Xueguang Lyu, Andrea Baisero, Yuchen Xiao, and Chris Amato. A deeper understanding of state-based critics in multi-agent reinforcement learning. In AAAI Conference on Artificial Intelligence, 2022. URL https://api.semanticscholar.org/CorpusID:245669036

    2022

  26. [34]

    Revisiting Plasticity in Visual Reinforcement Learning : Data , Modules and Training Stages

    Guozheng Ma, Lu Li, Sen Zhang, Zixuan Liu, Zhen Wang, Yixin Chen, Li Shen, Xueqian Wang, and Dacheng Tao. Revisiting Plasticity in Visual Reinforcement Learning : Data , Modules and Training Stages . October 2023

    2023

  27. [35]

    Python for Data Analysis: Data Wrangling with Pandas, NumPy , and IPython

    Wes McKinney. Python for Data Analysis: Data Wrangling with Pandas, NumPy , and IPython . O'Reilly Media, 1 edition, February 2013. ISBN 9789351100065. URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20&path=ASIN/1449319793

    2013

  28. [36]

    Spectral normalization for generative adversarial networks

    Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=B1QRgziT-

    2018

  29. [37]

    Rusu, Joel Veness, Marc G

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, ...

    2015 doi

  30. [38]

    Siddharth Mysore, Bassel El Mabsout, Renato Mancuso, and Kate Saenko. Honey. I Shrunk The Actor : A Case Study on Preserving Performance with Smaller Actors in Actor - Critic RL . In 2021 IEEE Conference on Games ( CoG ) , pp.\ 01--08, Copenhagen, Denmark, 2021. IEEE. ISBN 978...

    2021

  31. [39]

    Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning

    Michal Nauman, Michał Bortkiewicz, Piotr Miłoś, Tomasz Trzciński, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In Proceedings of the 41st International Conference on Machine Lear...

    2024

  32. [40]

    The Primacy Bias in Deep Reinforcement Learning

    Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The Primacy Bias in Deep Reinforcement Learning . In Proceedings of the 39th International Conference on Machine Learning . PMLR, June 2022

    2022

  33. [41]

    Oliphant

    Travis E. Oliphant. Python for scientific computing. Computing in Science & Engineering, 9 0 (3): 0 10--20, 2007. doi:10.1109/MCSE.2007.58

    2007 doi

  34. [42]

    The difficulty of passive learning in deep reinforcement learning

    Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net...

    2021

  35. [43]

    Puterman

    Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779

    1994

  36. [44]

    Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell

    Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation, 2016. URL https://arxiv.org/abs/1511.06295

    2016 arXiv

  37. [45]

    A Large Recurrent Action Model : xLSTM enables Fast Inference for Robotics Tasks , February 2025

    Thomas Schmied, Thomas Adler, Vihang Patil, Maximilian Beck, Korbinian Pöppel, Johannes Brandstetter, Günter Klambauer, Razvan Pascanu, and Sepp Hochreiter. A Large Recurrent Action Model : xLSTM enables Fast Inference for Robotics Tasks , February 2025

    2025

  38. [46]

    The dormant neuron phenomenon in deep reinforcement learning

    Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th Internation...

    2023

  39. [47]

    Maxinforl: Boosting exploration in reinforcement learning through information gain maximization

    Bhavya Sukhija, Stelian Coros, Andreas Krause, Pieter Abbeel, and Carmelo Sferrazza. Maxinforl: Boosting exploration in reinforcement learning through information gain maximization. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openr...

    2025

  40. [48]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the-book-2nd.html

    2018

  41. [49]

    Deep Reinforcement Learning for Robotics : A Survey of Real - World Successes

    Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martín-Martín, and Peter Stone. Deep Reinforcement Learning for Robotics : A Survey of Real - World Successes . November 2024

    2024

  42. [50]

    DeepMind Control Suite , January 2018

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind Control Suite , January 2018

    2018

  43. [51]

    dm\_control: Software and tasks for continuous control

    Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm\_control: Software and tasks for continuous control. Software Impacts, 6: 0 100022, November 2020. ISSN 2665-9638

    2020

  44. [52]

    L2 regularization versus batch and weight normalization

    Twan van Laarhoven. L2 regularization versus batch and weight normalization. CoRR, abs/1706.05350, 2017. URL http://arxiv.org/abs/1706.05350

    2017 arXiv

  45. [53]

    Python reference manual

    Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995

    1995

  46. [54]

    Wurman, and Peter Stone

    Miguel Vasco, Takuma Seno, Kenta Kawamoto, Kaushik Subramanian, Peter R. Wurman, and Peter Stone. A Super -human Vision -based Reinforcement Learning Agent for Autonomous Racing in Gran Turismo . 2024

    2024

  47. [55]

    Motion planning and control for mobile robot navigation using machine learning: a survey

    Xuesu Xiao, Bo Liu, Garrett Warnell, and Peter Stone. Motion planning and control for mobile robot navigation using machine learning: a survey. Autonomous Robots, 46 0 (5): 0 569--597, June 2022. ISSN 1573-7527

    2022

  48. [56]

    Benchmarking Reinforcement Learning Techniques for Autonomous Navigation

    Zifan Xu, Bo Liu, Xuesu Xiao, Anirudh Nair, and Peter Stone. Benchmarking Reinforcement Learning Techniques for Autonomous Navigation . In 2023 IEEE International Conference on Robotics and Automation ( ICRA ) , pp.\ 9224--9230, May 2023

    2023

  49. [57]

    Improving sample efficiency in model-free reinforcement learning from images, 2020

    Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images, 2020. URL https://openreview.net/forum?id=HklE01BYDB

    2020

  50. [58]

    Image augmentation is all you need: Regularizing deep reinforcement learning from pixels

    Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=GY6-6sTvGaf

    2021

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.