{"id":"3da725d6-8d6b-48c0-9463-2674bf823a5e","arxiv_id":"2501.12620","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adaptively lowering the number of per-episode update epochs via a multi-armed bandit improves or matches PPO/DrAC returns on standard benchmarks while reducing training FLOPS.","lead":"ADEPT adds a small slot-machine style scheduler to PPO and DrAC that chooses how many times to reuse each batch of experience for training, usually selecting fewer repetitions. It reports equal or better scores on Procgen, MiniGrid, and PyBullet while cutting floating-point operations by roughly 30 percent in several settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing fixed-epoch baseline (e.g., K=1) confounds the central claim: ADEPT(U) mostly selects K=1, so gains may reflect fewer gradient steps, not adaptation.","rationale":"The reader's verdict identified the value-prediction signal as the weakest assumption and also noted in the rationale that ADEPT(U)'s gain could stem from fewer gradient passes. I agree with the latter and believe it is the more load-bearing concern for the central claim. The value-prediction issue affects the quality of the bandit's decisions, but even a poorly calibrated bandit that mostly picks K=1 would reproduce the reported results if K=1 is a better fixed schedule than K=3. Therefore, the most decisive missing evidence is a fixed low-NUE baseline, which the paper does not provide. The paper's ablations vary hyperparameters and the NUE candidate set, but they never include PPO with NUE=1 fixed; the closest is the vanilla PPO with NUE=3. I locate this gap in the main experimental design of Section 5.2 and in the decision-process analysis of Figure 5. The Discussion's stated limitations are real but secondary; they do not address the confounding. The recommended verdict remains CONDITIONAL: the paper is plausible and has a well-executed experimental setup, but the central claim of adaptive benefit requires the missing control. If the fixed K=1 control supports ADEPT(U), the claim stands; if not, the contribution would be reduced to showing that fewer epochs can be beneficial, which is not the paper's stated novelty.","tokens_in":37906,"tokens_out":2908,"duration_ms":29625,"concrete_test":"Run PPO on the same 16 Procgen environments with fixed NUE=1, fixed NUE=2, and PPO+ADEPT(U) using identical hyperparameters, environment steps, and random seeds as in the paper (see Tables 1 and Appendix B.1). Compare normalized final test scores on the full distribution of levels and total training FLOPS using the paper's metrics. If fixed K=1 achieves statistically equivalent or higher scores than ADEPT(U) with comparable or lower compute, the central adaptation claim is falsified; if fixed K=1 performs worse, the claim survives this control. Also report the selection proportions to confirm that ADEPT(U) indeed chooses K=1 about 60% of the time.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that adaptive selection of the number of update epochs (NUE) improves performance and reduces compute. The key evidence is Figure 5, which shows PPO+ADEPT(U) selects K=1 about 60% of the time and K=2 or K=3 each about 20%, giving an average NUE around 1.6 versus the vanilla PPO schedule of NUE=3. Under this policy, ADEPT(U) outperforms vanilla PPO in 14 Procgen environments and achieves the lowest overhead in 11. However, no fixed-epoch control is reported: the paper does not compare against PPO with NUE=1 or NUE=2 under otherwise identical hyperparameters and seeds. If fixed K=1 attains the same or better final test scores and similar compute savings, then the observed gains are fully explained by reducing gradient steps—a known regularization effect—and the bandit's adaptive choices provide no measurable benefit. The Discussion (Section 6) concedes that value-prediction inaccuracies can degrade scheduling, but the more fundamental threat is that the schedule's output (mostly K=1) makes the adaptation itself epiphenomenal. The paper asserts in Section 4 that 'a dynamic NUE value can reduce reliance on specific data and preserve plasticity,' yet this assertion is untested against the natural static low-NUE baseline. Without that baseline, the headline claim of adaptive improvement is not supported; the results are confounded by the average epoch count.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ADEPT, a multi-armed bandit framework that adaptively selects the number of update epochs (NUE) per episode for on-policy RL algorithms (PPO and DrAC). Three schedulers are proposed: UCB (ADEPT(U)), Gaussian Thompson sampling (ADEPT(G)), and round-robin (ADEPT(R)). The method is evaluated on Procgen, MiniGrid, and PyBullet, with claims of improved data efficiency, better generalization, and reduced computational overhead. The central empirical claim is that PPO+ADEPT(U) outperforms vanilla PPO in 14 Procgen environments while using the least training FLOPS in 11. The paper also provides ablation studies, hyperparameter searches, and a FLOPS calculation appendix.","tokens_in":38199,"tokens_out":9271,"duration_ms":88848,"significance":"If the headline results held, ADEPT would be a simple, useful plug-in for reducing RL training compute while preserving or improving performance. The paper has tangible strengths: it releases code, evaluates on three benchmarks with multiple seeds, includes ablations, and uses proper aggregate metrics (mean, median, IQM, OG). However, the current evidence does not establish that adaptation, rather than a lower average number of update epochs, drives the gains. The missing fixed-epoch baselines are the decisive omission, and the FLOPS metric is defined in a way that makes the compute savings partially tautological. These issues are fixable within the scope of an empirical paper, but they currently prevent acceptance.","major_comments":[{"comment":"The headline performance claim is confounded by the absence of fixed-epoch baselines. Figure 5 shows that ADEPT(U) selects K=1 roughly 60% of the time, for an average NUE near 1.6, while the vanilla PPO comparison uses the fixed schedule NUE=3 (Section 5.1 and Appendix B.1). PPO+ADEPT(U) is reported to outperform vanilla PPO in 14 environments, but the paper never runs PPO with fixed K=1 or K=2 under the same hyperparameters and seeds. If fixed K=1 attains comparable or better test returns, the gains are explained by the regularization effect of fewer gradient steps, not by adaptive choice, and the claim in Section 4 that 'a dynamic NUE value can reduce reliance on specific data and preserve plasticity' is untested. The fixed low-NUE controls are essential to support Q1 and the abstract's claim of adaptive improvement.","section":"Section 5.2, Figure 5"},{"comment":"The computational-overhead result is largely a mechanical consequence of the FLOPS definition. O_forward is defined as proportional to N_update_epochs and O_backward = 2*O_forward, so total training FLOPS is linear in the average NUE. Because ADEPT(U) mostly chooses K=1 (Figure 5), its average NUE of ~1.6 mechanically yields an update-phase FLOPS ratio near 1.6/3 = 53% relative to vanilla PPO's NUE=3. The reported 70% total overhead (Section 5.2) follows from combining this with the constant sampling overhead. The authors should report the average NUE per scheduler and compare against fixed-NUE baselines at matched FLOPS and matched performance; without this, 'minimum computational overhead' does not demonstrate the benefit of adaptation.","section":"Appendix G, Eqs. (16)-(17)"},{"comment":"Hyperparameters are selected on the evaluation benchmark itself. The authors searched c ∈ {0.1, 1.0, 5.0} and W ∈ {10, 50, 100} on Procgen and then report the best configuration as the main result; the same procedure is used for PyBullet (Appendix B.3). This is test-set tuning, which inflates the reported gains and invalidates the claim that ADEPT is parameter-free. In addition, Figure 4's 'ADEPT(*)' bars take the best result among RRS, UCB, and GTS for each metric; this is a multiple-comparisons selection without correction. Please either fix the hyperparameters a priori, use a separate validation split, or report each variant's results and justify the 'best-of-three' aggregation.","section":"Appendices B.1 and F.1, Figure 4"},{"comment":"The Procgen 'data efficiency' comparison is plotted against FLOPS, not environment steps. Q1 asks whether ADEPT improves data efficiency, but the main training curves show performance per FLOP, which is a computational-efficiency measure; data efficiency in the sense of returns per environment step is not shown for Procgen (the MiniGrid and PyBullet figures do use environment steps). The paper should include environment-step curves to support the data-efficiency claim, or consistently use the term 'computational efficiency' for the Procgen results.","section":"Section 5.2, Figure 3"},{"comment":"The bandit's reward signal is the mean value-predicted return after K update epochs, but this value network is the same network being updated by the selected K. The Discussion concedes that inaccurate value predictions will directly affect scheduling quality, yet the paper provides no evidence that this self-referential estimate is reliable across tasks and learning stages (e.g., a correlation analysis between the bandit reward and final policy return). Without such evidence, the claim that ADEPT automatically selects the optimal extent value (Section 1) is not validated beyond the effect of lowering the average NUE.","section":"Equation (3) and Section 6"}],"minor_comments":[{"comment":"The initialization of µ and σ² for the Gaussian Thompson sampling arm is not specified; please report the initial values used in the experiments.","section":"Section 4.2, Eq. (5)"},{"comment":"The captions state that the dotted line and dashed line represent the highest score and the lowest overhead, respectively, but these appear as markers rather than lines; please clarify the visual convention.","section":"Figure 3 and Appendix D"},{"comment":"The formula for sampling overhead counts (Nepisode length + 1) observations per environment; note that this assumes a specific PPO implementation detail (the extra bootstrap prediction), and it should be stated explicitly that the reported FLOPS exclude environment-execution cost, which is acknowledged only in Figure 2(a).","section":"Appendix G, Eq. (14)"},{"comment":"The sentence 'Our method provides a systematic guarantee for optimizing long-term returns' overclaims, as no formal guarantee is derived; please rephrase to 'aims to optimize' or provide a proof.","section":"Section 2.1"},{"comment":"The description of the evaluation metrics says 'the average floating point operations (FLOPS) over 16 environments and all the runs'; consider clarifying that the FLOPS totals are per method and are then averaged, and specify the seed count in the main text rather than only in Appendix.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a preprint (arXiv:2501.12620) under review. The missing fixed-epoch baselines are the main scientific obstacle, and I expect the authors can add them with moderate effort (PPO and DrAC with NUE=1 and NUE=2 on the same environments and seeds). I also note that the 'best-of-three' aggregation and test-set hyperparameter search are practices that would need to be remedied in revision. The novelty relative to simply choosing fewer update epochs (e.g., a fixed K=1) is not yet demonstrated. If the authors add the controls and reframe the claims, the paper could become a useful empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: it is a solid, well-run RL study with a clear confound at its center. The idea is to let a multi-armed bandit pick the number of update epochs per episode for on-policy algorithms, and the authors test three schedulers—UCB, Thompson sampling, and round-robin—on PPO and DrAC across Procgen, MiniGrid, and PyBullet. They also release code and report FLOPS carefully.\n\nWhat is actually new is the control knob: bandits have been used to choose data augmentations, intrinsic rewards, and meta-controllers, but not the update-epoch count. That is a narrow but legitimate extension. The experimental work is thorough: five seeds, multiple benchmarks, ablations on every hyperparameter, decision-process plots, and a useful round-robin variant that serves as a partial control for adaptation.\n\nThe soft spot is load-bearing. Figure 5 shows the UCB variant selects K=1 about 60% of the time, so the average number of epochs is roughly 1.6 versus the vanilla 3. The paper never compares against a fixed NUE=1 baseline. If fixed K=1 does as well or better, then the reported gains come from doing fewer gradient steps, not from the bandit adapting. That would make the headline claim of adaptive improvement unsupported. The stress-test note is right: this is not a minor omission, it is the central comparison the paper needs.\n\nTwo smaller issues fit the same pattern. The FLOPS savings are partly definitional, since Appendix G counts training cost as linear in NUE, so a scheduler that often picks K=1 is guaranteed to report lower cost by that metric. And the hyperparameters are tuned on the same benchmark, with the ADEPT(*) aggregate cherry-picking the best of three variants. Neither kills the paper, but both push the headline numbers in the favorable direction.\n\nTo be fair, the paper is honest about limitations: the Discussion concedes value-prediction inaccuracy affects scheduling, and the code is available for others to check. The round-robin variant is a good sanity check, though it also lacks a fixed low-NUE comparison.\n\nBottom line: this deserves serious refereeing. The experiments are thorough, the plug-in is cheap, and practitioners will want to know whether the adaptive scheduler beats just setting K=1. A revision that adds fixed K=1 and K=2 baselines, and ideally equal-compute comparisons, would settle the question. As it stands, the adaptive component is plausible but unproven.","headline":"Bandit-scheduled update epochs is a sensible and well-tested idea, but without fixed K=1/K=2 baselines the paper cannot yet show that the adaptation itself helps.","tokens_in":38758,"tokens_out":2412,"would_cite":false,"duration_ms":27257,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the number of update epochs per episode in on-policy RL is a tunable resource that a multi-armed bandit can schedule online, and reports significant gains in data efficiency and compute.","keywords":["adaptive data exploitation","multi-armed bandit","number of update epochs","data efficiency","generalization","Proximal Policy Optimization","computational overhead","procedural generation"],"falsifier":"Run vanilla PPO with a fixed NUE of 1 on the same Procgen environments and compare its returns and measured FLOPs against PPO+ADEPT(U); if the fixed one-epoch schedule already matches the adaptive agent's performance and overhead, then the adaptive bandit is not what produces the gains. Conversely, corrupt the value-prediction reward signal with artificial noise and check whether scheduling quality degrades in step with the noise, which would confirm the loop's dependence on that signal.","tokens_in":37679,"feed_emoji":"🎰","tokens_out":10816,"duration_ms":94148,"temperature":0.7,"pith_summary":"The paper claims that the number of update epochs an on-policy RL algorithm spends reusing each collected episode is a tunable resource that should be scheduled online rather than fixed by hand. It introduces ADEPT, a multi-armed bandit that picks an update-epoch count for each episode using the episode's value-predicted return, and reports that this adaptive schedule matches or exceeds fixed three-epoch PPO on most Procgen environments while cutting measured training FLOPS to about 70% or less. The same wrapping improves DrAC, a data-augmented actor-critic, and the paper also reports faster learning on sparse-reward MiniGrid tasks and gains on PyBullet continuous control. If correct, ADEPT offers a plug-and-play way to make deep RL training cheaper, faster to adapt, and less prone to overfitting without adding any new learning objective.","feed_headline":"Adaptive update counts beat fixed schedules in 14 of 16 Procgen games","feed_subtitle":"Bandit scheduler beats fixed three-epoch PPO on 14 of 16 Procgen games while cutting update FLOPs to roughly 70%.","key_machinery":"The load-bearing object is the number of update epochs (NUE), the count of gradient steps the agent takes on a collected episode's experience. The carrying mechanism is a multi-armed bandit whose arms are candidate NUE values: the reward for selecting an arm is the episode's mean value-predicted return, averaged over a sliding FIFO window, and the selection rules are UCB, Gaussian Thompson sampling, or round-robin. This creates a self-referential loop in which the value network both produces the bandit's reward signal and is the network being trained for the chosen number of epochs.","core_discovery":"The paper's central claim is that data reuse per episode is itself a controllable resource that can be scheduled online. ADEPT frames each candidate number of update epochs (NUE) as an arm of a multi-armed bandit; after each episode it selects an arm, updates the policy and value networks for that many epochs, and feeds the mean value-predicted return of the episode into a sliding-window estimate of that arm's quality. Three schedulers are instantiated: UCB (ADEPT(U)), Gaussian Thompson sampling (ADEPT(G)), and round-robin (ADEPT(R)). On the sixteen Procgen environments, PPO+ADEPT(U) is reported to outperform vanilla PPO on 14 environments, reach the highest performance on 6, and produce the lowest computational overhead on 11, while DrAC+ADEPT(U) reaches the highest performance on 7 environments at about 69.1% of the overhead of vanilla DrAC. The paper interprets these results as evidence that different tasks and learning stages call for different amounts of data reuse, and that a lightweight bandit scheduler can find those amounts automatically.","pith_inferences":["The same bandit view could be applied to replay ratio or gradient steps per sample in off-policy RL, another hand-tuned reuse knob; this would be a direct transfer of the NUE mechanism to a different training loop.","Replacing the value-predicted return with the true episode return (available in simulation) would isolate how much of the gain comes from the scheduler itself rather than from the quality of its self-referential signal, an experiment the paper leaves open.","If ADEPT's generalization gains come from limiting data reuse, the effect should be weaker on a single fixed level than on the full procedural distribution, since the overfitting mechanism it targets is stronger under distribution shift; this is a testable prediction."],"forward_implications":["Practitioners can wrap PPO or DrAC with ADEPT and expect equal or better returns at roughly 70% of the update-phase FLOPs, without hand-tuning a fixed epoch count.","The agent regulates its own overfitting by switching to fewer update epochs when value predictions plateau, reducing repetition of the same batch and preserving plasticity.","Because ADEPT adds no auxiliary networks or extra loss terms, it can be combined with exploration and augmentation methods rather than competing with them.","The fixed three-epoch schedule reported as best for Procgen is not optimal at every learning stage, and an automatic scheduler can specialize the reuse count per environment."],"supporting_citations":[{"why":"Provides the Procgen benchmark and the fixed three-epoch NUE default used as the vanilla baseline.","marker":"(Cobbe et al., 2020)"},{"why":"PPO is the primary base algorithm that ADEPT wraps and compares against.","marker":"(Schulman et al., 2017)"},{"why":"Supplies the DrAC baseline and the earlier UCB-based bandit for automatic data augmentation that motivates treating RL subproblems as bandits.","marker":"(Raileanu et al., 2021)"},{"why":"The UCB algorithm that ADEPT(U) uses to select update-epoch counts.","marker":"(Auer, 2002)"},{"why":"The Thompson sampling rule behind ADEPT(G)'s Gaussian bandit.","marker":"(Thompson, 1933)"},{"why":"PPG, cited as evidence that different tasks benefit from different update strategies, motivating adaptive NUE.","marker":"(Cobbe et al., 2021)"},{"why":"The PPO implementation whose phase-wise FLOPS proportions motivate selecting the update phase as the target for overhead reduction.","marker":"(Huang et al., 2022)"}],"fun_headline_variants":["Bandit scheduler beats fixed update epochs on 14 of 16 Procgen games","ADEPT: bandit-driven update counts for data-efficient RL","Multi-armed bandit tunes data reuse in RL, reducing compute","Smarter data reuse: bandit picks update counts, wins on Procgen","Adaptive data exploitation: bandit scheduler cuts RL overhead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scheduling loop assumes the value network's predicted returns are reliable enough to rank the update-epoch counts, even though the same network is being updated a different number of times depending on what the bandit selects.","fun_headline_variants_meta":{"raw":{"variants":["Bandit scheduler beats fixed update epochs on 14 of 16 Procgen games","ADEPT: bandit-driven update counts for data-efficient RL","Multi-armed bandit tunes data reuse in RL, reducing compute","Smarter data reuse: bandit picks update counts, wins on Procgen","Adaptive data exploitation: bandit scheduler cuts RL overhead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000851,"raw_usage":{"total_tokens":3679,"prompt_tokens":900,"completion_tokens":2779,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":2700}},"tokens_in":516,"tokens_out":2779,"duration_ms":23712,"temperature":1.0,"reasoning_tokens":2700,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:59:15.021146+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run vanilla PPO with a fixed NUE of 1 on the same Procgen environments and compare its returns and measured FLOPs against PPO+ADEPT(U); if the fixed one-epoch schedule already matches the adaptive agent's performance and overhead, then the adaptive bandit is not what produces the gains. Conversely, corrupt the value-prediction reward signal with artificial noise and check whether scheduling quality degrades in step with the noise, which would confirm the loop's dependence on that signal.","supporting_citations":[{"cited_title":"W., Hilton, J., Klimov, O., and Schulman, J","cited_arxiv_id":null,"evidence_quote":"Provides the Procgen benchmark and the fixed three-epoch NUE default used as the vanilla baseline."},{"cited_title":"For the model update phase, the computational overhead is Oupdate = Oforward + Obackward, (15) where Oforward = Obs1 ∗ B ∗ Nbatches ∗ Nupdate epochs (16) and Obackward = Oforward ∗","cited_arxiv_id":null,"evidence_quote":"The PPO implementation whose phase-wise FLOPS proportions motivate selecting the update phase as the target for overhead reduction."}],"review_version":1}