Pith. sign in

REVIEW 5 major objections 5 minor 51 references

Online Training and Pruning of Deep Reinforcement Learning Networks

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reinforcement-learning agents using OFENet feature extractors can be pruned to about 40% of their initial size during training with only slight performance loss.

desk verdict Workmanlike extension of the authors' own XiNet pruning to OFENet-based SAC; the DenseNet-specific complexity cost is new, but the empirical case is under-powered and the forced rounding of theta at 80% training undermines the convergence claim. read the letter →

arxiv 2507.11975 v1 pith:5X5DUFPC submitted 2025-07-16 cs.LG cs.AIcs.RO

Valentin Frank Ingmar Guenter , Athanasios Sideris This is my paper
classification cs.LGcs.AIcs.RO
keywords ReinforcementlearningOnlinefeatureextractionNeuralnetworksStructuredpruningVariationalBernoulligatesSoftActor-CriticDenseNetMuJoCocontinuouscontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the cost of large neural networks in reinforcement learning can be paid once and then mostly recovered: start with an over-parameterized agent, prune it while it learns, and deploy a network a fraction of the original size. The proposed method, OFEXiNet, attaches a learnable 0/1 Bernoulli gate to every hidden unit of the OFENet feature extractor and of the policy, value, and Q networks of a Soft Actor-Critic agent, then trains gate probabilities and weights together. A complexity-aware regularization term, derived from the expected parameter and computational cost of the DenseNet feature extractor, automatically sets the pruning-strength hyperparameters and drives low-contribution units to zero. On MuJoCo continuous-control benchmarks the paper reports reductions to about 40% of initial size with slight performance loss, and finds that large networks pruned during training outperform same-size networks trained from scratch. If right, this gives a practical answer to the scaling dilemma of RL: keep the stability and performance of big networks, then shrink them before deployment.

What carries the argument

The central object is the learned gate: a Bernoulli random variable $\xi_k$ with variational parameter $\theta_k \in [0,1]$ multiplying the output of each hidden unit, converting each ordinary network into a XiNet. Sampling $\hat{\xi}$ at each step realizes a subnetwork for backpropagation, and the gate parameters are updated via projected gradient descent using $\partial C/\partial \xi_k$ as a straight-through estimator of the usefulness difference between keeping and removing a unit. The other load-bearing piece is the complexity-aware regularization $\Gamma(\Theta)$, chosen so that $\Gamma = \nu\,C(\Theta) + \mathrm{const}$, where $C(\Theta)$ is the expected parameter and computational complexity of the DenseNet feature extractor $\phi_o, \phi_{o,a}$ and the downstream policy and value networks, expressed as sums of products of $\|\theta_l\|_1$ terms. Layer-dependent coefficients make early DenseNet layers, whose outputs are concatenated and reused many times, cost more to keep, which is what steers pruning toward the units that most inflate deployment cost.

What would settle it

Rerun the Prune-B configuration on HalfCheetah-v2 and record the distribution of $\theta$ values when the final 20% rounding stage starts: if a large fraction of gates lie strictly between 0 and 1 at that point, the claimed convergence to a deterministic network has not occurred, and the reported final sizes depend on a manual rounding decision rather than on the gate dynamics.

Watch

Extended reading notes

Core claim

The paper's central claim, stated on its own terms, is that simultaneous training and pruning, already effective for supervised networks, can be integrated into modern RL algorithms, and that OFENet-enhanced SAC agents pruned this way retain nearly all of the large-network performance at a fraction of the parameter count. The mechanism is stochastic: each unit's output is scaled by a Bernoulli random variable $\xi_k \sim \mathrm{Bernoulli}(\theta_k)$, so every forward pass trains a randomly sampled subnetwork, while the variational parameters $\theta_k$ are updated by projected gradient descent with a straight-through gradient estimate. The optimization objective combines the RL losses and the OFENet's auxiliary next-state prediction loss with a regularization term that, through a flattening hyperprior, equals a scaled version of the expected parameter and computational complexity of the DenseNet feature extractor and the downstream networks. Because this regularization is multilinear in $\theta$, the paper argues the optimum sits at the extreme points of the unit hypercube, so gates end at 0 or 1 and the final network is deterministic: units with $\theta$ below tolerance are physically pruned during training. Empirically, the paper reports large OFENet networks reduced to about 40% of initial size with slight performance loss, and pruned agents that match or beat the standard SAC baseline with far fewer parameters.

Load-bearing premise

The load-bearing premise is that the gate probabilities $\theta$ actually converge to 0 or 1 while the agent is still learning, because only then can a pruned unit be treated as permanently dead; the paper itself notes that this convergence is observed in its experiments but sometimes at a reduced rate.

Editorial extensions

If this is right

  • Deploying an OFENet-based SAC agent becomes much cheaper: in several configurations the policy-side networks used at deployment shrink to well under half the baseline parameter count while keeping at least 95% of the baseline score.
  • Training is itself cheaper because each gradient step only backpropagates through the sampled sub-network, and units with $\theta$ below tolerance are physically removed as training proceeds.
  • The $\rho$ parameter gives an explicit dial between deployment-only compression and compression for continued learning, since only $\phi_o$ and the policy are needed once learning stops.
  • Starting from a large network and pruning during training yields higher-performing agents of a given final size than training a small network from scratch, mirroring the supervised-learning finding.
  • The same complexity-aware regularization can be carried over to convolutional feature extractors, since the cost formulas are written in terms of layer dimensions and gate norms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the $\theta \to \{0,1\}$ convergence survives non-stationary training, the practical recipe for RL scale is to over-parameterize for stability, prune online, and skip a separate prune-retrain pipeline; a direct test is to compare final performance when the rounding stage is removed and gates are left free.
  • The paper does not address what happens if the environment shifts after deployment; a prudential extension would reserve a small set of reactivatable units or re-open the gates periodically.
  • Because gate sampling acts like structured dropout, it may counteract the rank collapse observed in value networks during long training; measuring feature rank while pruning would show whether the gates are protecting representational capacity.
  • The dependence of the final architecture on the task, with Hopper pruning $\phi_{o,a}$ almost entirely while Cheetah keeps it wide, suggests the method is performing a form of architecture search; a testable follow-up is whether these per-environment patterns repeat across random seeds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes OFEXiNet, an integration of the authors' previous XiNet simultaneous training-and-pruning framework with the OFENet feature extractor for deep reinforcement learning. Bernoulli random variables gate individual units in the OFE networks and in the SAC policy/value networks, and a complexity-aware regularization term is designed so that, for a specific choice of per-layer prior parameters, the regularization matches the expected computational/parameter complexity of the DenseNet-style OFE architecture. The authors report experiments on HalfCheetah, Ant, Hopper, and Walker2D showing that OFENet-based SAC agents can be pruned to roughly half or less of their initial parameter count with modest performance loss, and that pruned large networks outperform smaller networks trained from scratch. The main claimed contributions are the architecture-specific complexity cost, the automatic selection of per-layer regularization hyperparameters, and the empirical demonstration of simultaneous RL training and structured pruning.

Significance. If the empirical claims are correct, the paper would be a useful step toward deploying large feature-extractor-based RL agents at lower memory and compute cost without a separate post-hoc pruning phase. The explicit expected-complexity formulas in Eqs. (19)-(23) are a concrete, architecture-aware contribution rather than a generic sparsity penalty, and the algorithmic integration with SAC is clearly specified. However, the load-bearing empirical claims are not yet established: the forced rounding of all unconverged gate parameters in the last 20% of training means the resulting deterministic networks are partly manufactured by the protocol rather than learned, the single-seed best-in-window scores are not adequate for the fine performance comparisons made, and one of the central prose claims about Table 3 is contradicted by the table's own numbers. The theoretical extreme-point argument is inherited from supervised learning and is not shown to transfer to the nonstationary RL objective. The paper does not provide code or multiple seeds, which further limits reproducibility.

major comments (5)
  1. [Section 3.1 and Section 5] The central convergence premise is unsupported because of the rounding protocol. Section 3.1 asserts that 'all θ values converge to either 0 or 1, albeit sometimes at a reduced rate,' but Section 5 states that 'for the last 20% of training, all θ parameters that have not yet completely converged to either 0 or 1 are rounded and fixed at 0 or 1.' Since the scores in Table 3 are measured after this forced rounding, the experiments cannot show that the deterministic pruned network is the learned result of the variational optimization; a substantial fraction of gates may be intermediate at the 80% mark. The supervised-learning extreme-point argument cited in Section 2.1 does not transfer to the nonstationary RL objective, and the flat gradient update in Eq. (6) does not by itself force convergence. Please report the fraction and value distribution of θ entries that are rounded at 80% of training, run at least one configuration without forced rounding to see whether gates converge on their own, and adjust the convergence claims accordingly.
  2. [Section 5, Table 3] The prose claim that 'for the other three environments' the method prunes parameters to at least half while maintaining at least 95% of the OFE-Big baseline score is contradicted by the table. For Prune-B, the scores relative to OFE-Big are 16033/17299 = 92.7% for HalfCheetah-v2, 6517/7608 = 85.7% for Ant-v2, and 5152/5490 = 93.8% for Walker2D-v2, all below 95%. For Prune-A, the deployment ratios in these environments are 87%, 130%, and 253%, which is not 'at least half' reduction. This contradiction directly affects the abstract and conclusion claims about reducing networks to 40% with 'only a slight performance loss.' Please either correct the claim or rerun with settings that actually satisfy the stated threshold.
  3. [Section 5, Table 3 and Figure 3] The experimental evaluation is based on a single seed and on the highest evaluation score observed during the last 20% of training. In continuous-control SAC benchmarks, seed-to-seed variance is large, and the best-in-window statistic inflates performance relative to a final-policy or mean-of-last-evaluations protocol. The margins in the 95% claims above (92.7%, 85.7%, 93.8%) are within the range that seed noise could reverse. Please report mean and standard deviation over at least five seeds, using a standard final-evaluation statistic, and state the number of seeds explicitly in the experimental setup.
  4. [Section 3.2, Eqs. (28)-(30)] The claim that the many hyperparameters associated with the flattening prior are 'automatically selected' is an overstatement. Eqs. (28) and (29) define log γ so that Eq. (30) holds, i.e., Γ_OFE = ν_OFE C_OFE + const; this is a reparameterization of the complexity cost, not a derivation of the hyperparameters from the optimization. The free hyperparameters in Table 2 still include ν_OFE, ν_π, ν_v, ν_q, λ_OFE, λ_RL, ρ, and θ_tol. The per-layer complexity weighting is a useful and concrete idea, but the 'automatic selection' language in the abstract and introduction should be replaced by an accurate description of what is actually automatic: the per-layer coefficients are no longer independent tuning knobs because they are determined by the architecture's expected complexity up to a small set of global scales.
  5. [Section 6 and Table 3] The conclusion that pruning large networks during training 'produces more efficient and higher performing RL agents rather than training smaller networks from scratch' is only clearly supported for deployment cost, not for training cost. Comparing Prune-B to the smaller SAC baseline in Table 3, the train ratios are 47% vs. 35% for HalfCheetah, 72% vs. 49% for Ant, and 88% vs. 44% for Walker2D, meaning Prune-B trains networks that are larger than SAC by this measure. If 'efficiency' refers only to the deployed policy, the conclusion should say so; if it includes training cost, the current evidence does not support it.
minor comments (5)
  1. [Section 5] There is a typo in 'various trafe-offs' and the text does not state the number of random seeds used; please add a complete experimental-protocol paragraph including environment version strings, number of seeds, and evaluation frequency.
  2. [Figure 3] The legend entries repeat method names multiple times (e.g., 'SACSAC', 'SAC-BigSAC'), which appears to be a plotting artifact and makes the figure difficult to read.
  3. [Section 2.1] The background section restates the authors' own prior work [26,35] at considerable length; a shorter self-contained summary would improve readability without changing the content.
  4. [Table 2] The regularization coefficients span several orders of magnitude (e.g., ν_v is 50 times ν_π in Prune-A); a sentence explaining the intended effect of this asymmetry would help readers interpret the sensitivity of the method.
  5. [Section 4] Related pruning methods for RL are discussed but never compared experimentally; even one comparison to a standard RL pruning baseline (e.g., gradual magnitude pruning as in [33]) would greatly strengthen the empirical claims.

Circularity Check

3 steps flagged · score 4.0 of 10

Partial circularity: the 'automatic' hyperparameter selection and the theta-convergence-to-0/1 claim are constructed by definition, and the deterministic-network guarantee imports the authors' own extreme-point theorem; the headline pruning results against external baselines retain independent content.

  1. self definitional [Section 3.2, Eqs. (28)-(30) (with Abstract claim of 'automatically selected')]
    "By taking in (13) log γ o l and log γ o,a l to satisfy for l = 1, . . . , Lo: ... we obtain Γ OFE = ν OFE · C OFE + const. (30) ... Thus, the regularization term Γ OFE (Θ o, Θ 0,a) in (12) matches the OFENet complexity C OFE within a scaling factor ν OFE > 0 that can be used to trade-off performance vs. complexity."

    The log γ hyperparameters are not selected by any fitting or optimization procedure; Eqs. (28)-(29) define them as explicit functions of the remaining θ parameters and architecture constants, chosen exactly so that (30) holds. The Abstract's claim that 'the many hyperparameters associated with them are automatically selected' therefore restates the definition: Γ_OFE is a scaled copy of C_OFE by construction, and the only free tuning knob is the single scalar ν_OFE. The matching in (30) is imposed, not derived, so this 'automatic selection' step is a reparameterization of the complexity cost rather than an independent result.

  2. uniqueness imported from authors [Section 3.1 (deterministic-network guarantee; also Section 2.1)]
    "In the supervised learning setting of [29, 35] and with a multilinear regularization function Γ OFE in Θ o, Θ o,a, the optimal solution of (14) is shown to be at one of extreme points of the hypercube H, corresponding to a deterministic pruned network. In the RL setting, where the distribution of the triplets ( o t, a t, o t+1) changes during the agent's learning, we still observe in our experiments that all θ values converge to either 0 or 1, albeit sometimes at a reduced rate."

    The premise that variational parameters converge to 0/1, so units can be 'rendered permanently inactive and pruned,' is the load-bearing guarantee of the method. That guarantee is imported from the authors' own prior work ([29, 35] are same-author preprints) as if it were an external theorem, and it is proved only in the supervised setting. The transfer to the non-stationary RL setting is not derived; the paper substitutes the empirical observation that 'all θ values converge to either 0 or 1.' Since the cited theorem is a self-citation whose RL transfer is only asserted, and since Section 5's rounding protocol actually enforces the extremes, the deterministic-network claim reduces to a self-citation chain rather than an independently established result.

1 more flagged steps
  1. fitted input called prediction [Section 5, evaluation protocol for Table 3 (convergence claim of Section 3.1)]
    "For the last 20% of training, all θ parameters that have not yet completely converged to either 0 or 1 are rounded and fixed at 0 or 1, so that at the end of training a deterministic network results. ... In this period, our XiNets have already fixed θ values at either 0 or 1 and hence have attained their final sizes and are deterministic."

    The Section 3.1 observation that 'all θ values converge to either 0 or 1' is made true by construction: the protocol pins every unconverged θ to an extreme in the last 20% of training. The convergence claim is therefore self-fulfilling rather than empirical evidence that the self-cited extreme-point theorem transfers to RL. Moreover, the headline numbers are produced only after this forced step: the final network sizes are attained once θ are fixed at extremes, and the Table 3 scores are measured during the last 20% of training when θ 'have already fixed' values.

full rationale

The paper's core empirical contribution—OFEXiNet prunes OFENet-based SAC agents to roughly 40% of their initial size with modest performance loss—is supported by head-to-head measurements against external baselines: plain SAC [9], SAC-Big, and OFE/OFE-Big from [16]. Scores, deploy ratios, and train ratios in Tables 3-4 are measured, not derived, so the central claim is not forced by the paper's equations. The explicit complexity counting in Eqs. (17)-(27) is a genuine derivation for the DenseNet concatenation structure and is not circular. Two premise-level steps are, however, circular in the weaker sense of being true by construction. First, Eqs. (28)-(30) define the log γ hyperparameters so that Γ_OFE = ν_OFE·C_OFE; the Abstract's 'automatically selected' is a reparameterization, not a selection procedure, leaving only the scalar ν_OFE to trade off performance versus complexity. Second, the deterministic-network guarantee is imported from the authors' own supervised-learning theorems [29,35] and does not transfer to the non-stationary RL setting; the paper substitutes an empirical claim ('all θ values converge to either 0 or 1') that Section 5's protocol forces by rounding and fixing every unconverged θ in the last 20% of training, with all final scores and sizes measured after that forced step. This makes the convergence premise self-fulfilling and partially conflates the protocol with the learned pruning. Because the headline pruning results are externally benchmarked rather than derived from the equations, the paper is not a derivation that reduces to its own inputs; the circularity is partial and premise-level, not result-level.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The method relies on several hand-chosen hyperparameters (nu, lambda, rho, theta_tol, initialization schedules) that control the sparsity/performance trade-off and are varied across Prune-A/B/C. The core pruning mechanism is inherited from the authors' prior variational Bernoulli gate formulation; the paper's new contribution is the complexity cost formulas for the DenseNet-based OFE network and the coupling of deployment-vs-training cost through rho.

free parameters (8)
  • nu_OFE = 5e-8, 2e-7, 3e-7 (Prune-A/B/C)
    Global scaling of the OFE complexity regularization in Eq. (33); hand-tuned per experiment to control pruning strength.
  • nu_pi = 2e-6, 1e-5, 2e-5
    Regularization strength for the policy network complexity in Eq. (34).
  • nu_v and nu_q = 1e-4, 5e-4, 2.5e-3
    Regularization strengths for value and Q networks.
  • lambda_OFE = 1e-9 or 1e-6
    L2 weight decay for OFE networks.
  • lambda_RL = 1e-9
    L2 weight decay for RL networks.
  • rho = 1.0 (Prune-A), 0.5 (Prune-B/C)
    Discount factor weighting deployment vs continued-learning complexity in Eq. (27).
  • theta_tol = 0.1
    Pruning tolerance; units with theta below this value are removed.
  • theta initialization and freezing schedule = theta=1 for first 200k steps; rounded in last 20%
    Hand-set schedule; affects which units survive.
assumptions (5)
  • domain assumption The flattening hyperprior on Bernoulli priors yields a linear-in-theta regularization whose minimizers are at theta = 0 or 1.
    Inherited from [26,35]; the paper relies on this to produce a deterministic pruned network. In RL with non-stationary data it is only observed empirically (Section 3.1).
  • domain assumption The Straight-Through estimator approximation C1_k - C0_k is approximately dC/dxi_k is accurate enough for updating theta.
    Used in Eq. (7) and Algorithm 1; if the approximation fails, the theta updates may prune the wrong units. The justification is cited from [26].
  • domain assumption All theta parameters converge to 0 or 1 during RL training, so permanent pruning is safe.
    Stated explicitly in Section 3.1: 'we still observe in our experiments that all theta values converge to either 0 or 1, albeit sometimes at a reduced rate'. No proof is given for the non-stationary case.
  • domain assumption The expected parameter-complexity expressions (21)-(24) are a faithful proxy for actual computational cost of the DenseNet OFE networks, including the diagonal pass-through weights.
    The formulas count floating-point operations under a specific convention; using them as a regularization target assumes they correlate with practical runtime.
  • domain assumption The auxiliary next-observation prediction task (8) yields features that are useful for the RL policy and value networks.
    Inherited from OFENet [16]; pruning based on this auxiliary task assumes that units unimportant for prediction are also unimportant for control.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Training and Pruning of Deep Reinforcement Learning Networks." pith.science (2026). https://pith.science/paper/5X5DUFPC

@misc{pith2026250711975,
  author       = {Pith},
  title        = {Pith review of: Online Training and Pruning of Deep Reinforcement Learning Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5X5DUFPC}},
  note         = {Machine review of arXiv:2507.11975}
}
abstract

Scaling deep neural networks (NN) of reinforcement learning (RL) algorithms has been shown to enhance performance when feature extraction networks are used but the gained performance comes at the significant expense of increased computational and memory complexity. Neural network pruning methods have successfully addressed this challenge in supervised learning. However, their application to RL is underexplored. We propose an approach to integrate simultaneous training and pruning within advanced RL methods, in particular to RL algorithms enhanced by the Online Feature Extractor Network (OFENet). Our networks (XiNet) are trained to solve stochastic optimization problems over the RL networks' weights and the parameters of variational Bernoulli distributions for 0/1 Random Variables $\xi$ scaling each unit in the networks. The stochastic problem formulation induces regularization terms that promote convergence of the variational parameters to 0 when a unit contributes little to the performance. In this case, the corresponding structure is rendered permanently inactive and pruned from its network. We propose a cost-aware, sparsity-promoting regularization scheme, tailored to the DenseNet architecture of OFENets expressing the parameter complexity of involved networks in terms of the parameters of the RVs in these networks. Then, when matching this cost with the regularization terms, the many hyperparameters associated with them are automatically selected, effectively combining the RL objectives and network compression. We evaluate our method on continuous control benchmarks (MuJoCo) and the Soft Actor-Critic RL agent, demonstrating that OFENets can be pruned considerably with minimal loss in performance. Furthermore, our results confirm that pruning large networks during training produces more efficient and higher performing RL agents rather than training smaller networks from scratch.

Figures

Figures reproduced from arXiv: 2507.11975 by the authors.

Figure 1
Figure 1. Schematic integration of the Online Feature Extractor Network [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. The Online Feature Extractor (OFE) network with DenseNet [39] architecture and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Cumulative rewards of the evaluation runs during training on the different environments. We [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The number of parameters in each found network using our method in thousands. Below each [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: The number of units in each layer of the OFEXiNets after training and pruning for [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 28 canonical work pages

  1. [35]

    Complexity-Aware Training of Deep Neural Networks for Optimal Structure Discovery

    V. F. I. Guenter and A. Sideris, “Complexity-aware training of deep neural networks for optimal structure discovery,” arXiv preprint arXiv:2411.09127 , 2024

    work page Pith review arXiv 2024

  2. [1]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems , vol. 25, 2012

    2012

  3. [2]

    Rethinking atrous convolution for semantic image segmentation,

    L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution for semantic image segmentation,” arXiv preprint arXiv:1706.05587 , 2017

    arXiv 2017

  4. [3]

    Speech recognition with deep recurrent neural networks,

    A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in 2013 IEEE international conference on acoustics, speech and signal processing , pp. 6645–6649, 2013

    work page 2013

  5. [4]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  6. [5]

    Self-supervised learning: Generative or contrastive,

    X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,” IEEE transactions on knowledge and data engineering , vol. 35, no. 1, pp. 857–876, 2021

    2021

  7. [6]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  8. [7]

    Playing atari with deep reinforcement learning,

    V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602 , 2013

    arXiv 2013

Show all 51 references
  1. [8]

    Human-level control through deep reinforce- ment learning,

    V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through dee...

    2015

  2. [9]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in International conference on machine learning , pp. 1861–1870, PMLR, 2018

    2018

  3. [10]

    Addressing function approximation error in actor-critic meth- ods,

    S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic meth- ods,” in International conference on machine learning , pp. 1587–1596, PMLR, 2018

    2018

  4. [11]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017

    2017 arXiv

  5. [12]

    Deep reinforcement learning and the deadly triad,

    H. Van Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil, “Deep reinforcement learning and the deadly triad,” arXiv preprint arXiv:1812.02648 , 2018. 22

    2018 arXiv

  6. [13]

    D2rl: Deep dense architectures in reinforcement learning,

    S. Sinha, H. Bharadhwaj, A. Srinivas, and A. Garg, “D2rl: Deep dense architectures in reinforcement learning,” arXiv preprint arXiv:2010.09163 , 2020

    2010 arXiv

  7. [14]

    What matters in on-policy reinforcement learning? a large-scale empirical study,

    M. Andrychowicz, A. Raichuk, P. Sta´ nczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, et al. , “What matters in on-policy reinforcement learning? a large-scale empirical study,” arXiv preprint arXiv:2006.05990 , 2020

    2006 arXiv

  8. [15]

    Deterministic policy gradient algorithms,

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in International conference on machine learning , pp. 387–395, PMLR, 2014

    2014

  9. [16]

    Can increasing input dimensionality improve deep reinforcement learning?,

    K. Ota, T. Oiki, D. Jha, T. Mariyama, and D. Nikovski, “Can increasing input dimensionality improve deep reinforcement learning?,” in International conference on machine learning , pp. 7424– 7433, PMLR, 2020

    2020

  10. [17]

    A framework for training larger networks for deep Reinforce- ment learning,

    K. Ota, D. K. Jha, and A. Kanezaki, “A framework for training larger networks for deep Reinforce- ment learning,” Machine Learning, vol. 113, pp. 6115–6139, Sept. 2024

    2024

  11. [18]

    Bigger, better, faster: human-level atari with human-level efficiency,

    M. Schwarzer, J. Obando-Ceron, A. Courville, M. G. Bellemare, R. Agarwal, and P. S. Castro, “Bigger, better, faster: human-level atari with human-level efficiency,” in Proceedings of the 40th International Conference on Machine Learning , ICML’23, 2023

    2023

  12. [19]

    Proto-value networks: Scaling representation learning with auxiliary tasks,

    J. Farebrother, J. Greaves, R. Agarwal, C. L. Lan, R. Goroshin, P. S. Castro, and M. G. Belle- mare, “Proto-value networks: Scaling representation learning with auxiliary tasks,” arXiv preprint arXiv:2304.12567, 2023

    2023 arXiv

  13. [20]

    Mastering diverse domains through world models,

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse domains through world models,” arXiv preprint arXiv:2301.04104 , 2023

    2023 arXiv

  14. [21]

    Learning both weights and connections for efficient neural networks,

    S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both weights and connections for efficient neural networks,” in International Conference on Neural Information Processing Systems - Volume 1, p. 1135–1143, 2015

    2015

  15. [22]

    What is the state of neural network pruning?,

    D. Blalock, J. J. Gonzalez Ortiz, J. Frankle, and J. Guttag, “What is the state of neural network pruning?,” in Machine Learning and Systems , vol. 2, pp. 129–146, 2020

    2020

  16. [23]

    Pruning filters for efficient convnets,

    H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” arXiv preprint arXiv:1608.08710 , 2016

    2016 arXiv

  17. [24]

    Lost in pruning: The effects of pruning neural networks beyond test accuracy,

    L. Liebenwein, C. Baykal, B. Carter, D. Gifford, and D. Rus, “Lost in pruning: The effects of pruning neural networks beyond test accuracy,” Proceedings of Machine Learning and Systems, vol. 3, pp. 93– 138, 2021

    2021

  18. [25]

    SCOP: scientific control for reliable neural network pruning,

    Y. Tang, Y. Wang, Y. Xu, D. Tao, C. Xu, C. Xu, and C. Xu, “SCOP: scientific control for reliable neural network pruning,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, pp. 10936–10947, Curran Associates Inc., Dec. 2020

    2020

  19. [26]

    Robust learning of parsimonious deep neural networks,

    V. F. I. Guenter and A. Sideris, “Robust learning of parsimonious deep neural networks,” Neuro- computing, vol. 566, p. 127011, 2024

    2024

  20. [27]

    Shallowing deep networks: Layer-wise pruning based on feature representa- tions,

    S. Chen and Q. Zhao, “Shallowing deep networks: Layer-wise pruning based on feature representa- tions,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 41, no. 12, pp. 3048– 3056, 2019. 23

    2019

  21. [28]

    Dbp: Discrimination based block-level pruning for deep model acceleration,

    W. Wang, S. Zhao, M. Chen, J. Hu, D. Cai, and H. Liu, “Dbp: Discrimination based block-level pruning for deep model acceleration,” arXiv preprint arXiv:1912.10178 , 2019

    1912 arXiv

  22. [29]

    Concurrent training and layer pruning of deep neural networks,

    V. F. I. Guenter and A. Sideris, “Concurrent training and layer pruning of deep neural networks,” arXiv preprint arXiv:2406.04549 , 2024

    2024 arXiv

  23. [30]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks,

    J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” in International Conference on Learning Representations , 2019

    2019

  24. [31]

    The state of sparse training in deep reinforcement learning,

    L. Graesser, U. Evci, E. Elsen, and P. S. Castro, “The state of sparse training in deep reinforcement learning,” in International Conference on Machine Learning , pp. 7766–7792, PMLR, 2022

    2022

  25. [32]

    Automatic noise filtering with dynamic sparse training in deep reinforcement learning,

    B. Grooten, G. Sokar, S. Dohare, E. Mocanu, M. E. Taylor, M. Pechenizkiy, and D. C. Mocanu, “Automatic noise filtering with dynamic sparse training in deep reinforcement learning,” in Proceed- ings of the 2023 International Conference on Autonomous Agents and Multiagent System...

    2023

  26. [33]

    In value-based deep reinforcement learning, a pruned network is a good network,

    J. Obando-Ceron, A. Courville, and P. S. Castro, “In value-based deep reinforcement learning, a pruned network is a good network,” in Proceedings of the 41st International Conference on Machine Learning, ICML’24, 2024

    2024

  27. [34]

    V. F. I. Guenter, Optimal Structure Discovery of Deep Neural Networks During Training and its Application to Reinforcement Learning. PhD thesis, UC Irvine, 2025

    2025

  28. [36]

    Observational overfitting in reinforcement learning,

    X. Song, Y. Jiang, S. Tu, Y. Du, and B. Neyshabur, “Observational overfitting in reinforcement learning,” arXiv preprint arXiv:1912.02975 , 2019

    1912 arXiv

  29. [37]

    A study on overfitting in deep reinforcement learning,

    C. Zhang, O. Vinyals, R. Munos, and S. Bengio, “A study on overfitting in deep reinforcement learning,” arXiv preprint arXiv:1804.06893 , 2018

    2018 arXiv

  30. [38]

    Learning state representation for deep actor-critic control,

    J. Munk, J. Kober, and R. Babuˇ ska, “Learning state representation for deep actor-critic control,” in 2016 IEEE 55th conference on decision and control (CDC) , pp. 4667–4673, IEEE, 2016

    2016

  31. [39]

    Densely connected convolutional net- works,

    G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional net- works,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700– 4708, 2017

    2017

  32. [40]

    Gymnasium: A standard interface for reinforcement learning environ- ments,

    M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goul˜ ao, A. Kallinteris, M. Krimmel, A. KG, et al. , “Gymnasium: A standard interface for reinforcement learning environ- ments,” arXiv preprint arXiv:2407.17032 , 2024

    2024 arXiv

  33. [41]

    Estimating or propagating gradients through stochastic neurons for conditional computation,

    Y. Bengio, N. L´ eonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432 , 2013

    2013 arXiv

  34. [42]

    Dropout: a simple way to prevent neural networks from overfitting,

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The journal of machine learning research , vol. 15, no. 1, pp. 1929–1958, 2014

    1929

  35. [43]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift,

    S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning , pp. 448–456, pmlr, 2015. 24

    2015

  36. [44]

    How does batch normalization help optimization?,

    S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, “How does batch normalization help optimization?,” Advances in neural information processing systems , vol. 31, 2018

    2018

  37. [45]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770–778, 2016

    2016

  38. [46]

    Deep reinforcement learning that matters,

    P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” in Proceedings of the AAAI conference on artificial intelligence , vol. 32, 2018

    2018

  39. [47]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning,

    Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in international conference on machine learning , pp. 1050–1059, PMLR, 2016

    2016

  40. [48]

    Dropout q-functions for doubly efficient reinforcement learning,

    T. Hiraoka, T. Imagawa, T. Hashimoto, T. Onishi, and Y. Tsuruoka, “Dropout q-functions for doubly efficient reinforcement learning,” in International Conference on Learning Representations , 2022

    2022

  41. [49]

    Regularization matters in policy optimization-an empirical study on continuous control,

    Z. Liu, X. Li, B. Kang, and T. Darrell, “Regularization matters in policy optimization-an empirical study on continuous control,” in International Conference on Learning Representations , 2020

    2020

  42. [50]

    Implicit under-parameterization inhibits data-efficient deep reinforcement learning,

    D. G. S. L. Aviral Kumar, Rishabh Agarwal, “Implicit under-parameterization inhibits data-efficient deep reinforcement learning,” in International Conference on Learning Representations, ICLR, 2021

    2021

  43. [51]

    A method for stochastic optimization,

    D. P. Kingma and J. Ba, “A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014. 25

    2014 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.