Adaptive Data Exploitation in Deep Reinforcement Learning

Bo Li; Mingqi Yuan; Wenjun Zeng; Xin Jin

REVIEW 5 major objections 5 minor 17 references

Adaptive Data Exploitation in Deep Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the number of update epochs per episode in on-policy RL is a tunable resource that a multi-armed bandit can schedule online, and reports significant gains in data efficiency and compute.

desk verdict Bandit-scheduled update epochs is a sensible and well-tested idea, but without fixed K=1/K=2 baselines the paper cannot yet show that the adaptation itself helps. read the letter →

arxiv 2501.12620 v1 pith:BPZWB572 submitted 2025-01-22 cs.LG cs.AI

Mingqi Yuan , Bo Li , Xin Jin , Wenjun Zeng This is my paper · ORCID

classification cs.LGcs.AI

keywords adaptivedataexploitationmulti-armedbanditnumberofupdateepochsefficiencygeneralizationProximalPolicyOptimizationcomputationaloverheadproceduralgeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the number of update epochs an on-policy RL algorithm spends reusing each collected episode is a tunable resource that should be scheduled online rather than fixed by hand. It introduces ADEPT, a multi-armed bandit that picks an update-epoch count for each episode using the episode's value-predicted return, and reports that this adaptive schedule matches or exceeds fixed three-epoch PPO on most Procgen environments while cutting measured training FLOPS to about 70% or less. The same wrapping improves DrAC, a data-augmented actor-critic, and the paper also reports faster learning on sparse-reward MiniGrid tasks and gains on PyBullet continuous control. If correct, ADEPT offers a plug-and-play way to make deep RL training cheaper, faster to adapt, and less prone to overfitting without adding any new learning objective.

What carries the argument

The load-bearing object is the number of update epochs (NUE), the count of gradient steps the agent takes on a collected episode's experience. The carrying mechanism is a multi-armed bandit whose arms are candidate NUE values: the reward for selecting an arm is the episode's mean value-predicted return, averaged over a sliding FIFO window, and the selection rules are UCB, Gaussian Thompson sampling, or round-robin. This creates a self-referential loop in which the value network both produces the bandit's reward signal and is the network being trained for the chosen number of epochs.

What would settle it

Run vanilla PPO with a fixed NUE of 1 on the same Procgen environments and compare its returns and measured FLOPs against PPO+ADEPT(U); if the fixed one-epoch schedule already matches the adaptive agent's performance and overhead, then the adaptive bandit is not what produces the gains. Conversely, corrupt the value-prediction reward signal with artificial noise and check whether scheduling quality degrades in step with the noise, which would confirm the loop's dependence on that signal.

Watch

Extended reading notes

Core claim

The paper's central claim is that data reuse per episode is itself a controllable resource that can be scheduled online. ADEPT frames each candidate number of update epochs (NUE) as an arm of a multi-armed bandit; after each episode it selects an arm, updates the policy and value networks for that many epochs, and feeds the mean value-predicted return of the episode into a sliding-window estimate of that arm's quality. Three schedulers are instantiated: UCB (ADEPT(U)), Gaussian Thompson sampling (ADEPT(G)), and round-robin (ADEPT(R)). On the sixteen Procgen environments, PPO+ADEPT(U) is reported to outperform vanilla PPO on 14 environments, reach the highest performance on 6, and produce the lowest computational overhead on 11, while DrAC+ADEPT(U) reaches the highest performance on 7 environments at about 69.1% of the overhead of vanilla DrAC. The paper interprets these results as evidence that different tasks and learning stages call for different amounts of data reuse, and that a lightweight bandit scheduler can find those amounts automatically.

Load-bearing premise

The scheduling loop assumes the value network's predicted returns are reliable enough to rank the update-epoch counts, even though the same network is being updated a different number of times depending on what the bandit selects.

Editorial extensions

If this is right

Practitioners can wrap PPO or DrAC with ADEPT and expect equal or better returns at roughly 70% of the update-phase FLOPs, without hand-tuning a fixed epoch count.
The agent regulates its own overfitting by switching to fewer update epochs when value predictions plateau, reducing repetition of the same batch and preserving plasticity.
Because ADEPT adds no auxiliary networks or extra loss terms, it can be combined with exploration and augmentation methods rather than competing with them.
The fixed three-epoch schedule reported as best for Procgen is not optimal at every learning stage, and an automatic scheduler can specialize the reuse count per environment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same bandit view could be applied to replay ratio or gradient steps per sample in off-policy RL, another hand-tuned reuse knob; this would be a direct transfer of the NUE mechanism to a different training loop.
Replacing the value-predicted return with the true episode return (available in simulation) would isolate how much of the gain comes from the scheduler itself rather than from the quality of its self-referential signal, an experiment the paper leaves open.
If ADEPT's generalization gains come from limiting data reuse, the effect should be weaker on a single fixed level than on the full procedural distribution, since the overfitting mechanism it targets is stronger under distribution shift; this is a testable prediction.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know about this paper: it is a solid, well-run RL study with a clear confound at its center. The idea is to let a multi-armed bandit pick the number of update epochs per episode for on-policy algorithms, and the authors test three schedulers—UCB, Thompson sampling, and round-robin—on PPO and DrAC across Procgen, MiniGrid, and PyBullet. They also release code and report FLOPS carefully.

What is actually new is the control knob: bandits have been used to choose data augmentations, intrinsic rewards, and meta-controllers, but not the update-epoch count. That is a narrow but legitimate extension. The experimental work is thorough: five seeds, multiple benchmarks, ablations on every hyperparameter, decision-process plots, and a useful round-robin variant that serves as a partial control for adaptation.

The soft spot is load-bearing. Figure 5 shows the UCB variant selects K=1 about 60% of the time, so the average number of epochs is roughly 1.6 versus the vanilla 3. The paper never compares against a fixed NUE=1 baseline. If fixed K=1 does as well or better, then the reported gains come from doing fewer gradient steps, not from the bandit adapting. That would make the headline claim of adaptive improvement unsupported. The stress-test note is right: this is not a minor omission, it is the central comparison the paper needs.

Two smaller issues fit the same pattern. The FLOPS savings are partly definitional, since Appendix G counts training cost as linear in NUE, so a scheduler that often picks K=1 is guaranteed to report lower cost by that metric. And the hyperparameters are tuned on the same benchmark, with the ADEPT(*) aggregate cherry-picking the best of three variants. Neither kills the paper, but both push the headline numbers in the favorable direction.

To be fair, the paper is honest about limitations: the Discussion concedes value-prediction inaccuracy affects scheduling, and the code is available for others to check. The round-robin variant is a good sanity check, though it also lacks a fixed low-NUE comparison.

Bottom line: this deserves serious refereeing. The experiments are thorough, the plug-in is cheap, and practitioners will want to know whether the adaptive scheduler beats just setting K=1. A revision that adds fixed K=1 and K=2 baselines, and ideally equal-compute comparisons, would settle the question. As it stands, the adaptive component is plausible but unproven.

Referee Report

5 major / 5 minor

Summary. The paper introduces ADEPT, a multi-armed bandit framework that adaptively selects the number of update epochs (NUE) per episode for on-policy RL algorithms (PPO and DrAC). Three schedulers are proposed: UCB (ADEPT(U)), Gaussian Thompson sampling (ADEPT(G)), and round-robin (ADEPT(R)). The method is evaluated on Procgen, MiniGrid, and PyBullet, with claims of improved data efficiency, better generalization, and reduced computational overhead. The central empirical claim is that PPO+ADEPT(U) outperforms vanilla PPO in 14 Procgen environments while using the least training FLOPS in 11. The paper also provides ablation studies, hyperparameter searches, and a FLOPS calculation appendix.

Significance. If the headline results held, ADEPT would be a simple, useful plug-in for reducing RL training compute while preserving or improving performance. The paper has tangible strengths: it releases code, evaluates on three benchmarks with multiple seeds, includes ablations, and uses proper aggregate metrics (mean, median, IQM, OG). However, the current evidence does not establish that adaptation, rather than a lower average number of update epochs, drives the gains. The missing fixed-epoch baselines are the decisive omission, and the FLOPS metric is defined in a way that makes the compute savings partially tautological. These issues are fixable within the scope of an empirical paper, but they currently prevent acceptance.

major comments (5)

[Section 5.2, Figure 5] The headline performance claim is confounded by the absence of fixed-epoch baselines. Figure 5 shows that ADEPT(U) selects K=1 roughly 60% of the time, for an average NUE near 1.6, while the vanilla PPO comparison uses the fixed schedule NUE=3 (Section 5.1 and Appendix B.1). PPO+ADEPT(U) is reported to outperform vanilla PPO in 14 environments, but the paper never runs PPO with fixed K=1 or K=2 under the same hyperparameters and seeds. If fixed K=1 attains comparable or better test returns, the gains are explained by the regularization effect of fewer gradient steps, not by adaptive choice, and the claim in Section 4 that 'a dynamic NUE value can reduce reliance on specific data and preserve plasticity' is untested. The fixed low-NUE controls are essential to support Q1 and the abstract's claim of adaptive improvement.
[Appendix G, Eqs. (16)-(17)] The computational-overhead result is largely a mechanical consequence of the FLOPS definition. O_forward is defined as proportional to N_update_epochs and O_backward = 2*O_forward, so total training FLOPS is linear in the average NUE. Because ADEPT(U) mostly chooses K=1 (Figure 5), its average NUE of ~1.6 mechanically yields an update-phase FLOPS ratio near 1.6/3 = 53% relative to vanilla PPO's NUE=3. The reported 70% total overhead (Section 5.2) follows from combining this with the constant sampling overhead. The authors should report the average NUE per scheduler and compare against fixed-NUE baselines at matched FLOPS and matched performance; without this, 'minimum computational overhead' does not demonstrate the benefit of adaptation.
[Appendices B.1 and F.1, Figure 4] Hyperparameters are selected on the evaluation benchmark itself. The authors searched c ∈ {0.1, 1.0, 5.0} and W ∈ {10, 50, 100} on Procgen and then report the best configuration as the main result; the same procedure is used for PyBullet (Appendix B.3). This is test-set tuning, which inflates the reported gains and invalidates the claim that ADEPT is parameter-free. In addition, Figure 4's 'ADEPT(*)' bars take the best result among RRS, UCB, and GTS for each metric; this is a multiple-comparisons selection without correction. Please either fix the hyperparameters a priori, use a separate validation split, or report each variant's results and justify the 'best-of-three' aggregation.
[Section 5.2, Figure 3] The Procgen 'data efficiency' comparison is plotted against FLOPS, not environment steps. Q1 asks whether ADEPT improves data efficiency, but the main training curves show performance per FLOP, which is a computational-efficiency measure; data efficiency in the sense of returns per environment step is not shown for Procgen (the MiniGrid and PyBullet figures do use environment steps). The paper should include environment-step curves to support the data-efficiency claim, or consistently use the term 'computational efficiency' for the Procgen results.
[Equation (3) and Section 6] The bandit's reward signal is the mean value-predicted return after K update epochs, but this value network is the same network being updated by the selected K. The Discussion concedes that inaccurate value predictions will directly affect scheduling quality, yet the paper provides no evidence that this self-referential estimate is reliable across tasks and learning stages (e.g., a correlation analysis between the bandit reward and final policy return). Without such evidence, the claim that ADEPT automatically selects the optimal extent value (Section 1) is not validated beyond the effect of lowering the average NUE.

minor comments (5)

[Section 4.2, Eq. (5)] The initialization of µ and σ² for the Gaussian Thompson sampling arm is not specified; please report the initial values used in the experiments.
[Figure 3 and Appendix D] The captions state that the dotted line and dashed line represent the highest score and the lowest overhead, respectively, but these appear as markers rather than lines; please clarify the visual convention.
[Appendix G, Eq. (14)] The formula for sampling overhead counts (Nepisode length + 1) observations per environment; note that this assumes a specific PPO implementation detail (the extra bootstrap prediction), and it should be stated explicitly that the reported FLOPS exclude environment-execution cost, which is acknowledged only in Figure 2(a).
[Section 2.1] The sentence 'Our method provides a systematic guarantee for optimizing long-term returns' overclaims, as no formal guarantee is derived; please rephrase to 'aims to optimize' or provide a proof.
[Section 5.1] The description of the evaluation metrics says 'the average floating point operations (FLOPS) over 16 environments and all the runs'; consider clarifying that the FLOPS totals are per method and are then averaged, and specify the seed count in the main text rather than only in Appendix.

Circularity Check

1 steps flagged · score 6.0 of 10

Compute-savings claim reduces to definition of FLOPS; fixed low-NUE baselines missing.

self definitional [Appendix G Eqs. (15)-(18); Section 4 key insight (iii); Section 5.2; Figure 5]
"O_forward = Obs1 ∗ B ∗ Nbatches ∗ N_update epochs ... O_total = (O_sampling + O_update) ∗ N_episodes (Appendix G). "By minimizing unnecessary updates through adaptive NUE tuning, we can significantly reduce the overall computational overhead" (Section 4). "ADEPT(U) primarily selects K = 1, while K = 2 and K = 3 each account for approximately 20%" (Figure 5). "Meanwhile, it produces the minimum computational overhead in 11 environments" (Section 5.2)."

The overhead metric is defined to be linear in N_update_epochs: O_forward = Obs1*B*Nbatches*N_update_epochs and O_total sums O_update per episode. Consequently, any scheduler whose average selected NUE is below the vanilla fixed-3 schedule is guaranteed to show lower measured FLOPS. Figure 5 shows ADEPT(U) chooses K=1 for roughly 60% of updates (K=2 and K=3 each about 20%), so its average NUE is about 1.6 versus 3 for vanilla PPO. The 'minimum computational overhead in 11 environments' is therefore an arithmetic consequence of the metric definition plus the scheduler's preference for K=1; it does not measure the value of adaptation. The FLOPS-savings 'prediction' is fixed by the definition of the metric itself.

full rationale

The central performance claim is not circular: ADEPT's returns are evaluated against environment scores on Procgen/MiniGrid/PyBullet, which are external to the scheduling mechanism, and the paper honestly reports learning curves and aggregate metrics. However, the computational-efficiency claim is a definitional artifact: Appendix G defines FLOPS so that update cost is proportional to N_update_epochs, and ADEPT(U) mostly selects K=1, so its lower measured overhead follows by construction rather than from adaptive scheduling. The paper also omits fixed NUE=1 and NUE=2 baselines, so the performance comparison cannot isolate adaptation from the regularizing effect of fewer gradient steps; this is a confound, though not a circular reduction. The Discussion's admitted reliance on value-predicted returns creates a self-referential bandit signal that is not validated against true returns, but the paper explicitly flags it. I found no load-bearing self-citations or imported uniqueness theorems: the authors' prior bandit work (AIRS) is cited only as context, and the NUE-candidate justification comes from external Procgen/DrAC papers. Overall, one advertised result (compute savings) reduces by construction, while the main empirical performance claim retains independent content, giving a partial circularity score of 6.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

ADEPT introduces no new physical or mathematical entities; its claims depend on standard bandit mathematics, on using value-network return estimates as the bandit reward, and on a FLOPS accounting that is linear in the number of update epochs.

free parameters (5)

UCB exploration coefficient c = 5.0 (Procgen), 1.0 (MiniGrid)
Chosen by grid search over {0.1, 1.0, 5.0}; controls how often the scheduler tries less-used NUE values.
Sliding window length W = 10 (UCB on Procgen), 50 (GTS on Procgen), 50 (MiniGrid)
Chosen per benchmark; W averages value-return estimates in Eq. (3).
GTS step size eta = 1.0 (PPO, Procgen), 0.1 (DrAC, Procgen)
Selected via search over {0.1, 0.5, 1.0}; updates the Gaussian mean in Eq. (5).
Initial GTS variance = unspecified
Eq. (4) requires sigma squared for each arm but the paper does not state its initialization.
NUE candidate set K = {3,2,1} for Procgen; {4,2,1}, {6,4}, {8,4} for MiniGrid; {10,5,1}, {10,5} for PyBullet
Defines the bandit arms; the paper varies K across benchmarks based on prior reported best NUE, and ablation shows larger K degrades overall performance.

assumptions (4)

standard math MAB algorithms (UCB, GTS) provide valid solutions for the NUE-selection problem
The paper treats NUE selection as a bandit problem in Section 4 without analyzing non-stationarity or regret.
domain assumption The value network's predicted return is a usable reward signal for scheduling
Eq. (3) uses sliding-window averages of estimated task returns; the Discussion admits inaccurate predictions hurt scheduling.
ad hoc to paper Arm rewards for GTS follow a normal distribution
Eq. (4) models each NUE's return as Gaussian; the Discussion acknowledges this may not generalize well.
domain assumption FLOPS of an update is forward FLOPS plus twice backward FLOPS and excludes environment execution
Appendix G defines O_backward = 2 * O_forward and counts only network-involved operations for overhead comparisons.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Adaptive Data Exploitation in Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/BPZWB572

@misc{pith2026250112620,
  author       = {Pith},
  title        = {Pith review of: Adaptive Data Exploitation in Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BPZWB572}},
  note         = {Machine review of arXiv:2501.12620}
}

read the original abstract

We introduce ADEPT: Adaptive Data ExPloiTation, a simple yet powerful framework to enhance the **data efficiency** and **generalization** in deep reinforcement learning (RL). Specifically, ADEPT adaptively manages the use of sampled data across different learning stages via multi-armed bandit (MAB) algorithms, optimizing data utilization while mitigating overfitting. Moreover, ADEPT can significantly reduce the computational overhead and accelerate a wide range of RL algorithms. We test ADEPT on benchmarks including Procgen, MiniGrid, and PyBullet. Extensive simulation demonstrates that ADEPT can achieve superior performance with remarkable computational efficiency, offering a practical solution to data-efficient RL. Our code is available at https://github.com/yuanmingqi/ADEPT.

Figures

Figures reproduced from arXiv: 2501.12620 by the authors.

**Figure 2.** Overview of the ADEPT framework. (a) The proportion of the computational overhead (FLOPS) is evaluated [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Training performance and computational overhead comparison of the PPO, DrAC, and their combinations with [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (33 more)

**Figure 4.** Figure 4: Aggregated performance of the PPO, DrAC, and their combinations with ADEPT on the test levels of the Procgen [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 6.** Figure 6: Aggregated training performance comparison of [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Performance of the PPO and its combinations with three ADEPT algorithms on the PyBullet benchmark. The [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: Aggregated performance of ADEPT(U) and ADEPT(R) on the MiniGrid benchmark. The mean and standard deviation are computed across all the environments. Data efficiency on MiniGrid. Additionally, we evaluate ADEPT on the MiniGrid benchmark with sparse-rewards and goal-ori…

**Figure 9.** Figure 9: Screenshots of the sixteen Procgen environments. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

**Figure 10.** Figure 10: Screenshots of the three MiniGrid environments. From left to right: [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 11.** Figure 11: Screenshots of the four PyBullet environments. From left to right: [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

**Figure 12.** Figure 12: Learning curves of the vanilla PPO agent and PPO+ADEPT(R). The mean and standard deviation are computed [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

**Figure 13.** Figure 13: Learning curves of the vanilla PPO agent and PPO+ADEPT(U) with different exploration coefficients. Here, the [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

**Figure 14.** Figure 14: Learning curves of the vanilla PPO agent and PPO+ADEPT(U) with different sizes of the sliding window. Here, [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]

**Figure 15.** Figure 15: Learning curves of the vanilla PPO agent and PPO+ADEPT(G) with different sizes of the sliding window. Here, [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]

**Figure 16.** Figure 16: Learning curves of the vanilla PPO agent and PPO+ADEPT(G) with different learning rates. Here, the size [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

**Figure 17.** Figure 17: Learning curves of the vanilla DrAC agent and DrAC+ADEPT(R). The mean and standard deviation are computed [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]

**Figure 18.** Figure 18: Learning curves of the vanilla DrAC agent and DrAC+ADEPT(U) with different exploration coefficients. Here, [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]

**Figure 19.** Figure 19: Learning curves of the vanilla DrAC agent and DrAC+ADEPT(U) with different sizes of the sliding window. [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]

**Figure 20.** Figure 20: Learning curves of the vanilla DrAC agent and DrAC+ADEPT(G) with different sizes of the sliding window. [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]

**Figure 21.** Figure 21: Learning curves of the vanilla DrAC agent and DrAC+ADEPT(G) with different learning rates. Here, the size [PITH_FULL_IMAGE:figures/full_fig_p025_21.png]

**Figure 22.** Figure 22: Performance and overhead comparison of the vanilla PPO agent and its combinations with ADEPT on the [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]

**Figure 23.** Figure 23: Performance and overhead comparison of the vanilla DrAC agent and its combinations with ADEPT on the [PITH_FULL_IMAGE:figures/full_fig_p027_23.png]

**Figure 24.** Figure 24: Detailed decision processes of PPO+ADEPT(U) on the Procgen benchmark. [PITH_FULL_IMAGE:figures/full_fig_p028_24.png]

**Figure 25.** Figure 25: Detailed decision processes of PPO+ADEPT(G) on the Procgen benchmark. [PITH_FULL_IMAGE:figures/full_fig_p029_25.png]

**Figure 26.** Figure 26: Detailed decision processes of DrAC+ADEPT(U) on the Procgen benchmark. [PITH_FULL_IMAGE:figures/full_fig_p030_26.png]

**Figure 27.** Figure 27: Detailed decision processes of DrAC+ADEPT(G) on the Procgen benchmark. [PITH_FULL_IMAGE:figures/full_fig_p031_27.png]

**Figure 28.** Figure 28: Aggregated training performance comparison of PPO+ADEPT(U) with different exploration coefficients and [PITH_FULL_IMAGE:figures/full_fig_p032_28.png]

**Figure 29.** Figure 29: Aggregated training performance comparison of PPO+ADEPT(G) with different learning rates and sizes of the [PITH_FULL_IMAGE:figures/full_fig_p032_29.png]

**Figure 30.** Figure 30: Aggregated training performance comparison of DrAC+ADEPT(U) with different exploration coefficients and [PITH_FULL_IMAGE:figures/full_fig_p033_30.png]

**Figure 31.** Figure 31: Aggregated training performance comparison of DrAC+ADEPT(G) with different learning rates and sizes of the [PITH_FULL_IMAGE:figures/full_fig_p033_31.png]

**Figure 32.** Figure 32: Aggregated training performance comparison of PPO+ADEPT(R) with different sets of NUE values. The mean [PITH_FULL_IMAGE:figures/full_fig_p034_32.png]

**Figure 33.** Figure 33: Aggregated training performance comparison of PPO+ADEPT(U) with different sets of NUE values. Here, [PITH_FULL_IMAGE:figures/full_fig_p035_33.png]

**Figure 34.** Figure 34: Aggregated training performance comparison of PPO+ADEPT(G) with different sets of NUE values. Here, the [PITH_FULL_IMAGE:figures/full_fig_p036_34.png]

**Figure 35.** Figure 35: Aggregated training performance comparison of DrAC+ADEPT(R) with different sets of NUE values. The mean [PITH_FULL_IMAGE:figures/full_fig_p037_35.png]

**Figure 36.** Figure 36: Aggregated training performance comparison of DrAC+ADEPT(U) with different sets of NUE values. Here, [PITH_FULL_IMAGE:figures/full_fig_p038_36.png]

**Figure 37.** Figure 37: Aggregated training performance comparison of DrAC+ADEPT(G) with different sets of NUE values. Here, the [PITH_FULL_IMAGE:figures/full_fig_p039_37.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

[1]

For the model update phase, the computational overhead is Oupdate = Oforward + Obackward, (15) where Oforward = Obs1 ∗ B ∗ Nbatches ∗ Nupdate epochs (16) and Obackward = Oforward ∗

∗ Nenvironments ∗ Obs1 (14) where +1 is for predicting the returns of the next observations at the end of the episode, as shown in the PPO implementation of CleanRL (Huang et al., 2022). For the model update phase, the computational overhead is Oupdate = Oforward + Obackward, (15) where Oforward = Obs1 ∗ B ∗ Nbatches ∗ Nupdate epochs (16) and Obackward = ...

work page 2022
[2]

(17) Here, B is the batch size, and the overhead ratio of a forward pass to a backward pass is 1:2 as suggested by (Dario & Danny, 2018). Finally, the total computational overhead is Ototal = (Osampling + Oupdate) ∗ Nepisodes (18) In the Procgen experiments, we have Obs1 = 528384FLOPS Nenvironments = 64 Nepisode length = 256 Nepisodes = 1525 B = 2048 Nbat...

work page 2018
[7]

Dropout: a simple way to prevent neural networks from overfitting

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958,

1929
[9]

Rewarding episodic visitation discrepancy for exploration in reinforcement learning

Yuan, M., Li, B., Jin, X., and Zeng, W. Rewarding episodic visitation discrepancy for exploration in reinforcement learning. In Deep RL Workshop NeurIPS 2022, 2022a. Yuan, M., Pun, M.-O., and Wang, D. R ´enyi state entropy maximization for exploration acceleration in reinforce- ment learning. IEEE Transactions on Artificial Intelli- gence, 2022b. Yuan, M....

work page 2022
[10]

39 Adaptive Data Exploitation in Deep Reinforcement Learning G

The mean and standard deviation are computed across all the environments. 39 Adaptive Data Exploitation in Deep Reinforcement Learning G. Calculation of Computational Overhead To compare the computational efficiency of the baseline algorithms and ADEPT, we utilize the floating point operations (FLOPS) as the KPI. Moreover, we only count the computational ...

work page 2020
[12]

For each Procgen environment, Table 2 lists the best augmentation method of DrAC as reported in (Raileanu & Fergus, 2021)

of DrAC for the experiments, and Table 1 lists the shared and fixed hyperparameters. For each Procgen environment, Table 2 lists the best augmentation method of DrAC as reported in (Raileanu & Fergus, 2021). The candidates of NUE are also set as K = {3, 2, 1} for all the experiments. For ADEPT(U) and ADEPT(G), we run the same hyperparameter search as the ...

work page 2021
[50]

Finally, Table 3 illustrates the PPO hyperparameters, which remain fixed throughout all the experiments. B.3. PyBullet Figure 11: Screenshots of the four PyBullet environments. From left to right: Ant, Hopper, HalfCheetah, and Walker2D. Finally, we perform the experiments on the PyBullet benchmark using the PPO implementation of (Kostrikov, 2018). Since (...

work page 2018
[255]

In this part, we use the official implementation (Raileanu et al.,

Reward normalization Yes for PPO, No for DrAC LSTM No Stacked frames No Environment steps 25000000 Episode steps 256 Number of workers 1 Environments per worker 64 Optimizer Adam Learning rate 5e-4 GAE coefficient 0.95 Action entropy coefficient 0.01 Value loss coefficient 0.5 Value clip range 0.2 Max gradient norm 0.5 Batch size 2048 Discount factor 0.99...

work page

Show all 17 references

[1988]

Master- ing visual continuous control: Improved data-augmented reinforcement learning

Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Master- ing visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2021a. Yarats, D., Kostrikov, I., and Fergus, R. Image aug- mentation is all you n...

2020
[2002]

Badia, A

doi: 10.1023/A:1013689704352. Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z. D., and Blundell, C. Agent57: Outperforming the atari human benchmark. In Inter- national conference on machine learning, pp. 507–517. PMLR, 2020a. Badia, A. P., Spre...

doi
[2015]

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

arXiv
[2016]

Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P

URL http: //arxiv.org/abs/1511.05952. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. Proceedings of the International Conference on Learning Representations,

arXiv
[2017]

The policy loss is defined as: Lπ(θ) = −Eτ ∼π [min (ρt(θ)At, clip (ρt(θ), 1 − ϵ, 1 + ϵ) At)] , (7) where ρt(θ) = πθ(at|st) πθold (at|st) , (8) and ϵ is a clipping range coefficient

is an on-policy algorithm that is designed to improve the stability and sample efficiency of policy gradient methods, which uses a clipped surrogate objective function to avoid large policy updates. The policy loss is defined as: Lπ(θ) = −Eτ ∼π [min (ρt(θ)At, clip (ρt(θ), 1 − ...

2015
[2020]

W., Hilton, J., Klimov, O., and Schulman, J

Cobbe, K. W., Hilton, J., Klimov, O., and Schulman, J. Phasic policy gradient. In International Conference on Machine Learning, pp. 2020–2027. PMLR,

2020
[2021]

and Bai, Y

Coumans, E. and Bai, Y . Pybullet, a python module for physics simulation for games, robotics and machine learn- ing. URL http://pybullet.org, 2016–2018. Dario, A. and Danny, H. Ai and compute. OpenAI blog,

2016
[2022]

Then we test the PPO agent with three ADEPT algorithms

reported the best K = 10, we set the NUE candidates as K = {10, 5, 1} and K = {10, 5}. Then we test the PPO agent with three ADEPT algorithms. Moreover, we run a similar hyperparameter search as the Procgen experiments and report the best results of each method. Table 3 illust...
[2023]

Lever- aging procedural generation to benchmark reinforcement learning

Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Lever- aging procedural generation to benchmark reinforcement learning. In International conference on machine learn- ing, pp. 2048–2056. PMLR,

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

For the model update phase, the computational overhead is Oupdate = Oforward + Obackward, (15) where Oforward = Obs1 ∗ B ∗ Nbatches ∗ Nupdate epochs (16) and Obackward = Oforward ∗

∗ Nenvironments ∗ Obs1 (14) where +1 is for predicting the returns of the next observations at the end of the episode, as shown in the PPO implementation of CleanRL (Huang et al., 2022). For the model update phase, the computational overhead is Oupdate = Oforward + Obackward, (15) where Oforward = Obs1 ∗ B ∗ Nbatches ∗ Nupdate epochs (16) and Obackward = ...

work page 2022

[2] [2]

(17) Here, B is the batch size, and the overhead ratio of a forward pass to a backward pass is 1:2 as suggested by (Dario & Danny, 2018). Finally, the total computational overhead is Ototal = (Osampling + Oupdate) ∗ Nepisodes (18) In the Procgen experiments, we have Obs1 = 528384FLOPS Nenvironments = 64 Nepisode length = 256 Nepisodes = 1525 B = 2048 Nbat...

work page 2018

[3] [7]

Dropout: a simple way to prevent neural networks from overfitting

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958,

1929

[4] [9]

Rewarding episodic visitation discrepancy for exploration in reinforcement learning

Yuan, M., Li, B., Jin, X., and Zeng, W. Rewarding episodic visitation discrepancy for exploration in reinforcement learning. In Deep RL Workshop NeurIPS 2022, 2022a. Yuan, M., Pun, M.-O., and Wang, D. R ´enyi state entropy maximization for exploration acceleration in reinforce- ment learning. IEEE Transactions on Artificial Intelli- gence, 2022b. Yuan, M....

work page 2022

[5] [10]

39 Adaptive Data Exploitation in Deep Reinforcement Learning G

The mean and standard deviation are computed across all the environments. 39 Adaptive Data Exploitation in Deep Reinforcement Learning G. Calculation of Computational Overhead To compare the computational efficiency of the baseline algorithms and ADEPT, we utilize the floating point operations (FLOPS) as the KPI. Moreover, we only count the computational ...

work page 2020

[6] [12]

For each Procgen environment, Table 2 lists the best augmentation method of DrAC as reported in (Raileanu & Fergus, 2021)

of DrAC for the experiments, and Table 1 lists the shared and fixed hyperparameters. For each Procgen environment, Table 2 lists the best augmentation method of DrAC as reported in (Raileanu & Fergus, 2021). The candidates of NUE are also set as K = {3, 2, 1} for all the experiments. For ADEPT(U) and ADEPT(G), we run the same hyperparameter search as the ...

work page 2021

[7] [50]

Finally, Table 3 illustrates the PPO hyperparameters, which remain fixed throughout all the experiments. B.3. PyBullet Figure 11: Screenshots of the four PyBullet environments. From left to right: Ant, Hopper, HalfCheetah, and Walker2D. Finally, we perform the experiments on the PyBullet benchmark using the PPO implementation of (Kostrikov, 2018). Since (...

work page 2018

[8] [255]

In this part, we use the official implementation (Raileanu et al.,

Reward normalization Yes for PPO, No for DrAC LSTM No Stacked frames No Environment steps 25000000 Episode steps 256 Number of workers 1 Environments per worker 64 Optimizer Adam Learning rate 5e-4 GAE coefficient 0.95 Action entropy coefficient 0.01 Value loss coefficient 0.5 Value clip range 0.2 Max gradient norm 0.5 Batch size 2048 Discount factor 0.99...

work page