Iterated Population Based Training with Task-Agnostic Restarts

Alexander Chebykin; Peter A. N. Bosman; Tanja Alderliesten

REVIEW 2 major objections 5 minor 2 references

A new PBT variant, IPBT, automatically adjusts its own step size via restarts and matches or beats tuned baselines on 8 tasks without per-task tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 22:39 UTC pith:RO7T3REE

load-bearing objection A solid, well-executed empirical PBT variant that mostly delivers on its out-of-the-box claim, with a disclosed but real selection-bias risk in its fixed hyperparameters. the 2 major comments →

arxiv 2511.09190 v2 pith:RO7T3REE submitted 2025-11-12 cs.LG cs.NE

Iterated Population Based Training with Task-Agnostic Restarts

Alexander Chebykin , Tanja Alderliesten , Peter A. N. Bosman This is my paper

classification cs.LG cs.NE

keywords Population Based TrainingHyperparameter OptimizationBayesian Optimizationstep sizerestartsshrink-perturblow-budget HPOreinforcement learning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

All Population Based Training (PBT) algorithms adjust hyperparameters during neural network training, but they share a hidden meta-hyperparameter: the step size, i.e., how many weight updates occur between hyperparameter adjustments. Previous work shows this step size strongly affects performance, yet no method exists for choosing it. This paper introduces Iterated PBT (IPBT), which automates the step size through restarts: a run proceeds only while a data-driven check shows performance is still improving; when it stagnates, IPBT restarts with shrink-perturbed weights, reinitialized hyperparameters from a time-varying Bayesian optimizer, and a doubled step size. On 8 image classification and reinforcement learning tasks, IPBT with one fixed configuration outperforms five earlier PBT variants run with untuned step sizes, and matches or beats them when their step sizes are tuned per task — while spending a quarter of the tuning compute. The paper argues that this makes IPBT the PBT variant of choice for low-budget hyperparameter optimization.

Core claim

The central claim is that the step size of a PBT run (the interval between hyperparameter updates) can be removed as a user-chosen meta-hyperparameter and controlled automatically. IPBT continues a run only while its smoothed, standardized performance is improving at a meaningful rate; otherwise it restarts: weights are shrink-perturbed, half the population is randomly reinitialized, half the hyperparameters come from a time-varying Bayesian optimizer, and the step size doubles. On CIFAR-10/100, Fashion-MNIST, TinyImageNet, and four Brax RL tasks, IPBT with a single fixed configuration outscores untuned PBT variants, random search, ASHA, and SMAC3, and statistically ties a per-task-tuned BG-

What carries the argument

The load-bearing object is the restart loop: IPBT starts with a step size of 1% of the budget and doubles it on each restart, so quick gains are exploited early and longer-horizon targets are reached later. Two fixed, task-agnostic criteria decide when to restart: smoothed, z-scored performance must have improved within t_patience outer steps, and by at least one standard deviation over the last t_interval steps. Information is carried across restarts by shrink-perturb (weights scaled by λ_shrink plus γ_perturb-scaled noise, half the population reinitialized) and by a restart-level time-varying Bayesian optimizer that evaluates initial hyperparameters by the best score of their descendants.

Load-bearing premise

The out-of-the-box claim rests on the assumption that the fixed IPBT defaults (patience 3, interval 15, shrink-perturb (0.2, 0.1), 1% initial step size), selected on CIFAR-10/100 and Humanoid, transfer to unseen tasks without re-tuning.

What would settle it

Hold out two tasks from the same families used here (e.g., CIFAR-10 to SVHN, Brax Humanoid to a different control task), run IPBT with its published defaults, and compare against a PBT baseline with one step size tuned per task; if IPBT's mean normalized score falls significantly below the tuned baseline, the no-per-task-changes claim fails. A simpler within-paper check: re-run the 4-task ablation suite with t_patience and t_interval varied beyond the tested values and see whether the chosen defaults sit on a performance plateau; if they do not, the stagnation detector is itself a sensitive kn

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

PBT-family HPO becomes usable out of the box: the step size, previously a decisive and hard-to-set meta-hyperparameter, is adjusted automatically within a single run of fixed budget.
In the tuned comparison, IPBT is statistically indistinguishable from BG-PBT with per-task tuned step size while running once instead of four times, so the tuning cost can be spared without sacrificing expected performance.
IPBT automatically discovers hyperparameter schedules with restarts for any hyperparameter (learning rate, weight decay, augmentation strength), not just those with known warm-restart benefits.
Ablations show each component matters: exponential step-size growth, shrink-perturb weight reuse, random reinitialization of half the population, and time-varying BO reinitialization of hyperparameters all contribute to the IQM; removing any of them degrades results.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the fixed defaults were selected on two of the eight evaluation tasks (as the appendix reports), the strongest form of the out-of-the-box claim remains untested on genuinely unseen tasks.
The restart-level machinery could be pushed further: making shrink-perturb strengths and the detection thresholds themselves adaptive would dissolve the last fixed settings and give a fully parameter-free PBT.
Because IPBT discovers schedules rather than static hyperparameters, it connects naturally to warm-restart and continual-learning ideas; shrink-perturb may also act as a reset that mitigates catastrophic forgetting in lifelong learning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A solid, well-executed empirical PBT variant that mostly delivers on its out-of-the-box claim, with a disclosed but real selection-bias risk in its fixed hyperparameters.

read the letter

Colleague,

Quick take: IPBT is a genuinely useful empirical contribution to the PBT/HPO subfield, and the central claim mostly holds, but the "out-of-the-box" wording overstates what is actually shown. I'd send it to review.

What's new: the algorithm is a plausible recombination of known pieces—BG-PBT restarts, shrink-perturb, time-varying BO—but the specific mechanism of doubling the step size on data-driven restarts with task-agnostic weight reuse is new and addresses a real pain point: the step-size meta-HP that all PBT variants currently have. The empirical work is solid: 8 tasks, 8 seeds, bootstrap IQM with Holm-corrected tests, ablations, code and configs released. The comparison against untuned PBT variants is strongly in IPBT's favor, and even against per-task tuned PBT variants it is competitive while using less compute. The ablation study gives good evidence that the components matter, and the replacement of BG-PBT's distillation with shrink-perturb on RL tasks is a fair head-to-head that supports the task-agnostic claim.

Soft spots: the biggest one is the selection bias in the fixed hyperparameters. Appendix D admits that t_patience, t_interval, shrink-perturb values, and initial step size were chosen after preliminary runs on CIFAR-10/100 and Humanoid—three of the eight evaluation tasks. The authors use different seeds for preliminary vs. main runs, which prevents seed-level overfitting, but not task-level selection bias. If those HPs are tuned to the mean behavior of those tasks, the "no changes to its HPs" claim is weaker than "no changes across tasks used to select them." I don't think this breaks the paper—the ablations show robustness to some of these choices (e.g., 0.5% vs 1% initial step), and the margin over most baselines is large—but a held-out task evaluation or a robustness analysis over the fixed HPs would substantially strengthen it.

Second, the comparison to BG-PBT is underwhelming: IPBT matches but does not beat tuned BG-PBT, and the difference is not significant. That's fine, but it tempers the "preferred PBT variant" conclusion.

Third, SMAC3's poor performance is partly explained by budget-usage issues they worked around; that's a legitimate fix, but it means the baseline comparison is a bit apples-to-oranges relative to how a user would actually run SMAC3.

Bottom line: this paper is for researchers in dynamic HPO and PBT; it gives them a strong new baseline and a useful study of restart mechanisms. It deserves a serious referee, and the main thing to ask of the authors is held-out validation of the fixed hyperparameters. I'd cite it if I were working on PBT variants.

Referee Report

2 major / 5 minor

Summary. The paper introduces IPBT, a Population Based Training (PBT) variant that aims to remove the need to tune the PBT step-size meta-hyperparameter. IPBT runs PBT-like iterations; upon detecting stagnation via two data-driven criteria (a patience check and a standardized improvement check), it restarts the population, reuses weights via shrink-perturb and random reinitialization, reinitializes hyperparameters via time-varying Bayesian optimization, and doubles the step size on each restart. The method is evaluated on 8 tasks (4 image classification, 4 Brax RL) against 5 previous PBT variants and against random search, ASHA, and SMAC3, with 8 seeds, IQM/CI estimates, Holm-corrected bootstrap tests, and an ablation study. The headline claim is that with a single fixed set of hyperparameters, IPBT matches tuned BG-PBT and significantly outperforms the other baselines, using the same budget.

Significance. If the claim holds, IPBT is a practical contribution: it addresses the well-documented sensitivity of PBT variants to the step-size meta-hyperparameter and offers a strong out-of-the-box HPO method for low-budget settings. The empirical methodology has notable strengths: open-source code release, 8 tasks, 8 seeds, multiple-testing correction, separate seeds for development and final evaluation, and systematic ablations. The main caveat is that the fixed hyperparameters of IPBT were selected using a subset of the evaluation benchmark, so the out-of-the-box claim is not fully supported by the current experiments as stated.

major comments (2)

[Appendix D; §4.1] The central 'out-of-the-box' claim is weakened because the fixed hyperparameters were selected on tasks that appear in the evaluation set. Appendix D states that t_patience=3 was chosen by comparing 2, 3, and 4 on CIFAR-10/100 and Humanoid; t_interval=15 was set by visual inspection of preliminary traces; shrink-perturb (0.2, 0.1) was chosen after trying (0.5, 0.5) on CIFAR-10; and the initial step size 1% was selected after ablations on CIFAR-10/100, Humanoid, and Hopper. Three of these four tasks are in the 8-task benchmark. Separate seeds do not prevent task-level selection bias. Please either validate on held-out tasks that were not touched during development, or substantially qualify the abstract and conclusion (e.g., 'without per-task changes among the benchmark tasks'). This is load-bearing for the headline claim.
[§4.1, Appendix C] The description of untuned PBT variants is ambiguous: 'the results across all steps are pooled' appears to mean 'across all step sizes.' If so, each untuned PBT baseline has four runs per seed (one per step size), while IPBT has one run per seed. The paired stratified bootstrap described in Appendix C samples seeds and uses them as indices for both algorithms; it is not clear how the multiple step-size runs are paired or aggregated. Please clarify the unit of analysis and, if pooling multiple runs per seed, either define a per-seed aggregation or use an appropriate blocked/unpaired test. This affects the significance claims in Figure 4.

minor comments (5)

[§3.2] The sentence 'standardization is done based on the entire trajectory' could be read as using future information when deciding a restart. Please clarify that the z-score normalization and GP smoothing are computed causally from data available at the decision point; if a full-trajectory standardization is used, that would be a lookahead issue.
[§4.1] The phrase 'results across all steps are pooled' should be 'across all step sizes' to avoid confusion with training steps.
[Figure 6] The caption says the 'best score of each variant is framed in red,' but the displayed numbers are IQM values, not individual scores. Please align the terminology.
[§4.3, Figure 7] Ablation claims such as 'worsen substantially' are based on IQM/CI overlap. Formal significance tests, or at least a statement about CI overlap, would make the ablation conclusions more robust.
[Appendix E] The SMAC3 fork that changes the stopping criterion is disclosed, but the paper should note explicitly that results may not be directly reproducible with stock SMAC3 and that this could affect comparisons with other published SMAC3 results.

Circularity Check

0 steps flagged

No circularity found: IPBT is an empirical combination of independently published components; the paper's claims do not reduce to fitted constants or self-citations.

full rationale

This paper makes no formal derivation or prediction claim that could collapse into its inputs. The algorithm is a stated combination of existing components (PBT, BG-PBT's BO and restarts, shrink-perturb from Ash and Adams) plus a new data-driven restart criterion and step-size doubling. The headline comparisons are against standard external baselines (random search, ASHA, SMAC3) and established PBT variants on public tasks; even though the PBT baselines come from the authors' own PBT-Zoo, that is an implementation/benchmark choice, not a by-construction equivalence. The self-cited observation that step size matters (Chebykin et al. 2025) is independently re-supported in this paper by Figure 6, so the motivation does not rest only on self-citation. Appendix D does disclose that IPBT's own fixed hyperparameters (t_patience=3, t_interval=15, shrink-perturb (0.2,0.1), initial step size 1%) were chosen after preliminary runs and visual inspection on CIFAR-10/100 and Humanoid, which are part of the evaluation set; this is a genuine benchmark-selection-bias risk for the 'out of the box' generalization claim, but it is not circularity by construction: the selected meta-parameters do not encode or force the reported normalized performance values, and the main experiments use different seeds and include external baselines. No equation, fitted parameter, or self-citation chain was found that reduces a claimed result to its own input.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 0 invented entities

The central empirical claim depends on a set of hand-picked algorithm hyperparameters fixed across tasks (listed as free parameters). The method also assumes that GP smoothing of within-run scores, meta-BO over initial HPs, shrink-perturb weight reuse, and step doubling are reasonable modeling choices for low-budget HPO; these are domain assumptions rather than proven results. No new physical or conceptual entities are introduced.

free parameters (8)

t_patience (restart patience) = 3
Number of consecutive non-improving outer steps before a restart is triggered; chosen after trying 2, 3, 4 on CIFAR-10/100 and Humanoid in preliminary experiments (Appendix D).
t_interval (improvement window) = 15
Window for the second restart criterion; set based on visual inspection of performance traces and not varied (Appendix D).
shrink-perturb parameters = lambda_shrink=0.2, gamma_perturb=0.1
Weight-reuse perturbation coefficients; default used in main experiments, varied in ablations; (0.5,0.5) was tried during development and performed worse (Appendix D).
random weight reinitialization fraction = 0.5
Half of the population has weights randomly reinitialized after a restart to escape bad basins (Section 3.3).
random HP reinitialization probability = 0.5
Half of the population receives random HPs after a restart (Section 3.3).
initial step size = 1% of budget
Start with small step size and double on each restart; 0.5% and 3% were tested in an extra ablation (Appendix D, Figure 9).
population multiple = 2
Start with 2N networks and keep the best N after the first outer step (Section 3.1); ablations show 1 and 3 perform worse.
step size growth factor = 2 (doubling)
Step size is doubled on each restart; linear increase is worse and constant step size is worst (Section 4.3).

axioms (5)

domain assumption Parametric GP regression with heteroscedastic noise can meaningfully smooth and standardize within-run performance for stagnation detection.
Section 3.2; the restart criteria depend on GP-smoothed z-scored scores being informative. If trajectories are highly non-stationary or noisy, the criteria may trigger incorrectly.
domain assumption Time-varying meta-BO over initial HPs, using the maximum descendant score as long-term performance, is a good model for choosing initial HPs at each restart.
Section 3.3; this assumes smooth dependence of long-term performance on initial HPs, which is not proven.
domain assumption Shrink-perturb preserves useful weight information while injecting enough noise to escape basins of attraction.
Section 3.3; this is based on prior work and ablations, not a proven property for all tasks.
ad hoc to paper Doubling the step size on each restart improves budget allocation (frequent updates early, longer intervals later).
Section 3.4; exponential growth is a heuristic. Ablations show it beats linear or constant growth, but there is no theoretical grounding.
domain assumption The tasks and setup of Chebykin et al. (2025) form a representative benchmark for low-budget HPO.
Section 4.1; the paper reuses the benchmark from the authors' prior work, so conclusions are limited to these 8 tasks.

pith-pipeline@v1.3.0-alltime-deepseek · 15729 in / 13320 out tokens · 120026 ms · 2026-08-03T22:39:07.230279+00:00 · methodology

0 comments

read the original abstract

Hyperparameter Optimization (HPO) can lift the burden of tuning hyperparameters (HPs) of neural networks. HPO algorithms from the Population Based Training (PBT) family are efficient thanks to dynamically adjusting HPs every few steps of the weight optimization. Recent results indicate that the number of steps between HP updates is an important meta-HP of all PBT variants that can substantially affect their performance. Yet, no method or intuition is available for efficiently setting its value. We introduce Iterated Population Based Training (IPBT), a novel PBT variant that automatically adjusts this HP via restarts that reuse weight information in a task-agnostic way and leverage time-varying Bayesian optimization to reinitialize HPs. Evaluation on 8 image classification and reinforcement learning tasks shows that, on average, our algorithm matches or outperforms 5 previous PBT variants and other HPO algorithms (random search, ASHA, SMAC3), without requiring a budget increase or any changes to its HPs. The source code is available at https://github.com/AwesomeLemon/IPBT.

Figures

Figures reproduced from arXiv: 2511.09190 by Alexander Chebykin, Peter A. N. Bosman, Tanja Alderliesten.

**Figure 1.** Figure 1: An example run of IPBT on the Humanoid task. Three iterations can be seen. Within each iteration, HPs are dynamically [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Figure 2: General PBT loop: in each outer step, the weights [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Visualizations of the smoothed standardized scores [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Figure 4: Normalized performance across 8 tasks (IQM and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 7.** Figure 7: Ablation studies of IPBT: normalized performance [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗

**Figure 10.** Figure 10: Extra ablation study on the weight reuse mecha [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗

**Figure 9.** Figure 9: Extra ablation study on the initial step size of IPBT: [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

2 extracted references · 1 linked inside Pith

[1979]

The results in Table 4 show that performance of IPBT is significantly different from that of all algorithms except for BG-PBT

is used to control the family-wise error rate. The results in Table 4 show that performance of IPBT is significantly different from that of all algorithms except for BG-PBT. D Hyperparameters The hyperparameters of all the algorithms are listed in the configuration files included in the source code release. In this section, we discuss the setting of hyper...

2025
[2021]

Freeman, C

OpenReview.net. Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. (2021). Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation. InProceed- ings of the Thirty-fifth Conference on Neural Information Processing Systems. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for Image Recog...

Pith/arXiv arXiv 2021

[1] [1979]

The results in Table 4 show that performance of IPBT is significantly different from that of all algorithms except for BG-PBT

is used to control the family-wise error rate. The results in Table 4 show that performance of IPBT is significantly different from that of all algorithms except for BG-PBT. D Hyperparameters The hyperparameters of all the algorithms are listed in the configuration files included in the source code release. In this section, we discuss the setting of hyper...

2025

[2] [2021]

Freeman, C

OpenReview.net. Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. (2021). Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation. InProceed- ings of the Thirty-fifth Conference on Neural Information Processing Systems. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for Image Recog...

Pith/arXiv arXiv 2021