Pith. sign in

REVIEW 4 major objections 5 minor 24 references

Hyperparameter brittleness in offline goal-conditioned RL is largely a consequence of bootstrapped value objectives, not an unavoidable property of RL.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:12 UTC pith:M4GQN4NW

load-bearing objection A careful, honestly hedged empirical study of hyperparameter landscape geometry in offline GCRL; worth refereeing once the arXiv abstract is corrected and the phase-boundary artifact is addressed. the 4 major comments →

arxiv 2602.05459 v2 pith:M4GQN4NW submitted 2026-02-05 cs.LG

Beyond Success Rates: Trainability and Extractability for Offline GCRL

Jan Malte T\"opperwien , Aditya Mohan , Marius Lindauer This is my paper
classification cs.LG
keywords offline reinforcement learninggoal-conditioned RLhyperparameter sensitivityhyperparameter landscapesTD bootstrappingquasimetric representation learninggradient interferencephase-resolved analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether hyperparameter sensitivity in deep RL is intrinsic or amplified by particular training mechanisms, and uses offline goal-conditioned RL to separate the two. It shows that, once modest expert data is present, a non-bootstrapped quasimetric objective produces broad, stable near-optimal hyperparameter regions, while a bootstrapped TD objective yields sharp optima that drift across training phases. The authors attribute the divergence to inter-goal gradient interference: bootstrapped targets make critic updates from different goal relabelings conflict more strongly, and the heaviness of that conflict correlates with landscape brittleness. The implication is that sensitivity can be designed away by choosing objectives that avoid recursive target coupling, reducing the need for phase-specific tuning schedules.

Core claim

In offline goal-conditioned RL with fixed datasets and controlled data quality, the choice of value-learning objective changes not just peak success but the entire geometry and transferability of the hyperparameter performance landscape. QRL, which learns a quasimetric distance-to-goal without bootstrapping, keeps a large fraction of configurations near-optimal and keeps its important hyperparameters stable across phases once about 20% expert data is present. HIQL, which learns a value function via bootstrapped TD expectile regression, shows tighter optima, stronger interactions between learning rate and discount factor, and larger phase-to-phase drift, especially under scheduled improvement

What carries the argument

The central instrument is the phased hyperparameter-landscape analysis paired with an inter-goal gradient-alignment diagnostic. Landscapes are built by training a fixed set of hyperparameter configurations from shared checkpoints, evaluating at phase boundaries, and summarizing geometry with ε-optimality mass (the fraction of configurations within, say, 90% of the best), phase-to-phase drift (mean normalized performance change), and early-selection regret (loss from committing to a configuration chosen in an early phase). The mechanistic diagnostic computes, for a single minibatch and multiple goal relabelings, the cosine similarity between critic-loss gradients; negative values indicate des

Load-bearing premise

The phase boundaries used to define training stages are derived from when a single hand-tuned default configuration first reaches 95% of its final success rate, and since QRL and HIQL converge at very different speeds, the phases may compare the two algorithms at different effective training progress — so the reported drift differences could be partly an artifact of boundary placement.

What would settle it

On a fixed offline dataset, take a quasimetric objective and add a bootstrapped target term to its critic loss: if the inter-goal gradient cosine-similarity distribution gains a heavier negative tail and the hyperparameter landscape becomes sharper and more phase-sensitive (lower ε-optimality mass, higher phase-to-phase drift), the paper's causal story is supported; if those metrics stay unchanged, the attribution of brittleness to bootstrapping is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Offline goal-conditioned RL hyperparameter landscapes are generally more benign than online RL landscapes: even with degraded data, large fractions of configurations stay near-optimal, and most landscape reorganization happens early in training.
  • With at least modest expert data, non-bootstrapped quasimetric objectives can make narrow, phase-specific hyperparameter schedules unnecessary, since near-optimal regions stay broad and stable.
  • Bootstrapped TD methods require more careful and stage-dependent tuning, and early good configurations can become suboptimal under scheduled data-quality shifts.
  • Inter-goal gradient-alignment measurements offer a low-cost proxy for predicting which objectives or configurations will be brittle: a heavier negative tail in critic gradient cosine similarity co-occurs with lower performance and sharper optima.
  • Objective design could target reducing cross-goal gradient conflicts while retaining bootstrapping's sample-efficiency benefits.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the gradient-interference story generalizes, any relabeling-based TD learner (including hindsight experience replay in online settings) should show the same phase-dependent brittleness; measuring cosine similarity between gradient updates from different relabelings in those settings is a direct test.
  • The comparison is observational, but it suggests a verifiable causal claim: adding a bootstrapped target term to QRL's critic loss should shift its gradient-alignment distribution toward a heavier negative tail and sharpen its landscape, while removing the bootstrap term from HIQL should do the reverse.
  • The phase-boundary calibration could be checked by recomputing drift and ε-optimality mass with boundaries aligned at equal training steps or equal performance progress; if the HIQL-versus-QRL gap shrinks, part of the reported effect is an artifact of the protocol.
  • The gradient-alignment diagnostic could be turned into an online monitor: track the negative-tail mass during training and trigger a learning-rate adjustment when it grows, effectively replacing hand-tuned phase schedules with a signal-driven schedule.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The full text of the manuscript (title: "When Are RL Hyperparameters Benign? A Study in Offline Goal-Conditioned RL") studies whether hyperparameter sensitivity in offline goal-conditioned RL is intrinsic to the problem or amplified by the training objective. It compares the bootstrapped TD method HIQL with the non-bootstrapped quasimetric method QRL under controlled stationary and scheduled data-quality regimes. Using phase-resolved hyperparameter landscapes, fANOVA importance profiles, and an inter-goal gradient-alignment diagnostic, it reports that (i) offline GCRL landscapes are generally more benign than online RL landscapes, (ii) once modest expert data is present, QRL exhibits broad and stable near-optimal regions while HIQL shows sharper optima and larger phase-to-phase drift, and (iii) HIQL's critic gradients show a heavier negative tail of inter-goal cosine similarity, which co-occurs with the observed sensitivity. The paper frames the gradient result as a descriptive diagnostic rather than a proof of causality.

Significance. If the reported findings hold, they would be a useful contribution to AutoRL and offline GCRL: they suggest that the choice of learning objective, not just the data distribution or exploration, can shape the geometry of hyperparameter response surfaces and that non-bootstrapped objectives may be substantially easier to tune. The study is computationally extensive (128 configurations, 5 seeds per phase, multiple AntMaze variants), defines quantitative landscape metrics (ε-optimality mass, drift, early-selection regret), and includes explicit limitations. The gradient-interference diagnostic is an interesting mechanistic hypothesis that is clearly distinguished from an optimality criterion. However, the paper's central cross-algorithm comparison is built on only two algorithms that differ in many dimensions beyond bootstrapping, and the phase-alignment protocol introduces a potential confound (Appendix D.1, Table 10). The manuscript also contains a severe internal inconsistency: the abstract supplied at the top describes a different paper (four methods, AWR temperature, Cube/Scene) from the full text (two methods, learning rate/discount factor, AntMaze variants). As it stands, the su

major comments (4)
  1. [Abstract vs. Full Text] The abstract states that the paper studies four methods (GCIQL, GCIVL, QRL, CRL) under a shared AWR extractor, with landscapes over learning rate and AWR temperature, evaluated on AntMaze, Cube, and Scene. The full text studies two methods (HIQL, QRL) over learning rate and discount factor, on AntMaze variants and related mazes, with no AWR temperature or Cube/Scene experiments. These are irreconcilably different studies. The reader cannot determine which claims are actually being made. This must be resolved before any further review.
  2. [Appendix D.1 / Table 10] Phase boundaries are defined from the last time a single hand-tuned default configuration crosses 95% of its final success rate. For antmaze-medium-v0 at 0% explore, this occurs at 18,461 steps for QRL and 99,734 steps for HIQL, a >5x difference. The four phases therefore cover very different absolute training windows for the two algorithms. The paper acknowledges this in D.1 but dismisses it as 'not problematic' because the goal is only to find interesting time steps. That dismissal is not valid for cross-algorithm comparisons of drift, near-optimal mass, and importance stability: if QRL's default is relatively well-tuned or its 95% bar is reached quickly, QRL's landscapes are sampled at systematically earlier absolute steps. The reported differences may be artifacts of the phase protocol rather than properties of the objectives. The authors should provide a sensitivity analysis with al
  3. [Tables 2–3 / Section 4.2] The headline quantitative differences are not statistically robust. For example, scheduled HIQL ρ0.9 = 0.12±0.10 vs. QRL 0.35±0.25 overlap within one standard deviation; across-phase change for HIQL 0.22±0.16 vs. QRL 0.17±0.11 also overlap. Moreover, HIQL and QRL differ in objective, architecture, regularization, and update rules, so the specific attribution to bootstrapping is not supported by the design. The limitations section correctly notes this, but the abstract and conclusion present the contrast as a finding. A concrete test would be a bootstrapped vs. non-bootstrapped variant of the same algorithm family, or an explicit statement that the paper only reports a two-point comparison.
  4. [Appendix E / Section 4.1] fANOVA importances are computed from SMAC/Hypersweeper optimization traces, not from a uniform design (Appendix E). This makes the importance vectors design-dependent: SMAC's proposal distribution changes over time and differs by algorithm. The phase-to-phase cosine distances in Table 1 and the perplexity/stability plots in Figure 2 could therefore reflect differences in the optimization trajectory rather than intrinsic shifts in which hyperparameters matter. The authors should either recompute importances on the fixed Sobol configuration set or justify why the non-uniform design does not affect the cross-algorithm comparisons.
minor comments (5)
  1. [Throughout] The full text contains multiple typos and inconsistencies in references/citations: 'Howeer' (Appendix A), 'Adavantage', 'quaismetric', 'explciitly', 'ICL' truncated citations, and incomplete venue entries. A careful proofread is needed.
  2. [Table 4] Table 4 reports phase-to-phase change aggregated over both algorithms, despite the text discussing algorithm-specific differences. Please split by algorithm or clarify that the table is a pooled summary.
  3. [Section 3 / Eq. (7)–(8)] The gradient diagnostic κ(g,g') uses gradients with respect to the critic trunk. It would help to state explicitly how 'critic trunk' is defined for HIQL and QRL, and whether the comparison is on the same parameter dimensionality. The Pearson correlation of 0.43 in Section 4.3 should also report sample size and confidence interval.
  4. [Figure 1 / Caption] The caption says 'top-10%' near-optimal region, while the text and Eq. (4) define ε-optimality mass with ε thresholds. Please use consistent terminology and indicate which ε is shown.
  5. [Appendix F.1] The definition of 'last intersection with the performance threshold' should specify interpolation details and how ties are handled, since phase boundary placement is load-bearing for the conclusions.

Circularity Check

0 steps flagged

No significant circularity; the empirical comparison and diagnostics are measured independently of the conclusions.

full rationale

The paper is an empirical landscape study, not a derivation, and its central comparisons do not reduce to their inputs by construction. Phase boundaries (Appendix D.1) are set from a single default configuration's 95% convergence point; the paper explicitly warns that 'this approach may yield phases that vary widely across algorithms' and that results 'should therefore not be used to compare convergence speed across algorithms or datasets.' That is a real protocol limitation for cross-algorithm comparability, but the landscape metrics (Eqs. 4-6) are not algebraically determined by the phase-boundary choice: rho_epsilon, Delta, and regret are measured from actual configuration evaluations in each phase. The gradient-interference diagnostic (Eqs. 7-8) is computed directly from critic gradients under each algorithm's own loss; it is not fitted to reproduce the landscape pattern, and the authors explicitly call it 'a descriptive mechanism consistent with the observed patterns rather than a definitive explanation' and state that 'cross-algorithm comparisons alone do not establish causality.' The heavy citation of Mohan et al. (2023) supplies the phased-pipeline methodology and IGPR visualization, not the paper's load-bearing claim, and the algorithms, datasets, and HPO tools (OGBench, SMAC, DeepCAVE) are external. No equation in the paper is equivalent by construction to an input, and no fitted parameter is renamed as a prediction. The D.1 phase-calibration limitation and the Section 4.3 causal-identification caveat are correctly acknowledged; they lower confidence in the strength of the HIQL-vs-QRL comparison but are not circularity. Verdict: no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No free parameters are fitted to produce the central pattern; the analysis is empirical. The main load-bearing assumptions are the comparability of the two objectives under goal relabeling, the phase-boundary protocol, and the validity of the gradient diagnostic as a measure of destructive interference.

axioms (5)
  • domain assumption Offline datasets with goal relabeling provide a standardized learning signal that allows fair comparison of different objectives on identical task structure (Section 1, Preliminaries).
    The entire comparison treats HIQL and QRL as comparable solely through the universal goal-reaching signal; if relabeling interacts differently with the two objectives, the comparison is confounded.
  • ad hoc to paper Phase boundaries defined by last crossing of 95% of final success rate of a hand-tuned default configuration align 'crucial points' across algorithms (Appendix D.1).
    The paper acknowledges the default config may be suboptimal and that phase lengths vary substantially; all landscape metrics depend on these phase definitions.
  • domain assumption Inter-goal gradient cosine similarity on critic parameters is a valid diagnostic of destructive interference for both objectives (Section 3, Eq. 7-8).
    The diagnostic compares different loss geometries (expectile TD vs quasimetric+regularization); high alignment can also mean weak goal dependence, which the paper acknowledges.
  • domain assumption Hyperparameter landscapes in offline GCRL are representative of RL tuning difficulty generally (title, abstract, Section 5).
    Generalization from two algorithms and maze tasks to 'RL hyperparameters are benign' is an extrapolation; the paper itself restricts scope in Limitations.
  • standard math fANOVA variance decomposition and Sobol sampling assumptions (Hutter et al. 2014; Sobol 1967) hold for the evaluated configuration sets.
    Used to compute importances and sample configurations; importances depend on the empirical design distribution induced by SMAC, which the paper notes.

pith-pipeline@v1.3.0-alltime-deepseek · 24762 in / 9922 out tokens · 102362 ms · 2026-08-03T04:12:42.276313+00:00 · methodology

0 comments
read the original abstract

Offline goal-conditioned reinforcement learning (GCRL) is typically benchmarked by the best tuned success rate of each method. This score measures attainable performance, but it does not reveal how reliably a learned goal-conditioned signal can be extracted into a policy: a method could succeed across many value-learning and extraction settings, or only at a narrow, hard-to-find configuration. We study this gap across four methods, GCIQL, GCIVL, QRL, and CRL, under a shared advantage-weighted regression (AWR) extractor. For each method, we construct trainability landscapes over the optimizer learning rate, which affects value learning and actor optimization, and AWR temperature, which controls how selectively the actor imitates high-advantage transitions. Across AntMaze, Cube, and Scene, we observe distinct regimes: high-scoring methods may be broadly accessible or brittle, while broad relative basins may still sit below low absolute ceilings. To interpret these differences, we pair landscapes with post-hoc diagnostics of future-vs-random goal discrimination and AWR weight concentration. Their relationship to downstream success is task-dependent. On AntMaze, where future goals align with path-like progress, these diagnostics explain landscape regimes. On Cube and Scene, goal ranking and manipulation control decouple: methods can rank goals well while failing downstream, or succeed through action-conditioned advantages despite weak future-vs-random separation. These results show that peak tuned success alone does not establish broadly extractable goal-conditioned behavior. Trainability landscapes expose this gap, while extraction diagnostics offer a lower-cost lens on how learned signals become policies.

Figures

Figures reproduced from arXiv: 2602.05459 by Aditya Mohan, Jan Malte T\"opperwien, Marius Lindauer.

Figure 1
Figure 1. Figure 1: Isolation of objective sensitivity to hyperparameter configuration changes. By using offline GCRL, we isolate how learning mechanisms respond to data quality without the confound of exploration noise. We visualize the near-optimal region (top-10%) in the learning-rate – discount-factor plane (HIQL on the left and QRL on the right). Top: Scheduled training phases (1 → 4) with improving data quality. Bottom:… view at source ↗
Figure 2
Figure 2. Figure 2: Phase-wise concentration and stability of fANOVA importances. We compare hyperparameter influence across scheduled data quality phases. Left Column (a, c): Perplexity of the normalized importance distribution; lower values indicate importance concentrates on fewer hyperparameters. Right Column (b, d): Importance-weighted Kendall’s τ between consecutive phases; higher values indicate a more stable ordering … view at source ↗
Figure 4
Figure 4. Figure 4: Phased training (Mohan et al., 2023). At t0 we start training all sampled configurations and evaluate them at tls(1). For the second phase, we start training from tls(1), loading the checkpoint of the best configuration from phase one, and then re-evaluate all configurations at tls(2). The best configuration is determined by evaluating at tfinal, which we will set to tls(i). fANOVA computation and normaliz… view at source ↗
Figure 5
Figure 5. Figure 5: QRL: Phase-wise fANOVA main-effect importances under scheduled data quality for antmaze-medium-v0. We define 100% training progress as the last time step at which the evaluation metric exceeds 95% of the final observed performance. Using the last intersection, rather than the first, avoids early transient effects and yields a more stable phase placement. The phase boundaries tls(i) are then set as fixed fr… view at source ↗
Figure 6
Figure 6. Figure 6: HIQL: Phase-wise fANOVA main-effect importances under scheduled data quality for antmaze-medium-v0. (a) Phase 1 (100% explore) (b) Phase 2 (80% explore) (c) Phase 3 (40% explore) (d) Phase 4 (0% explore) [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: HIQL: Phase-wise fANOVA main-effect importances under scheduled data quality for antmaze-large-v0. F.6. Phase Mobility Plots Phase mobility of near-optimal hyperparameters. We visualize how the optimal region shifts across training phases and data regimes. For each setting (scheduled or constant), we identify the "near-optimal" region as the set of hyperparameters achieving the top 10% of performance withi… view at source ↗
Figure 8
Figure 8. Figure 8: QRL: Phase-wise fANOVA main-effect importances under scheduled data quality for antmaze-large-v0. (a) Phase 1 (b) Phase 2 (c) Phase 3 (d) Phase 4 [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: QRL: Phase-wise fANOVA main-effect importances under expert data quality for antmaze-medium-v0. (a) Phase 1 (b) Phase 2 (c) Phase 3 (d) Phase 4 [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: HIQL: Phase-wise fANOVA main-effect importances under expert data quality for antmaze-medium-v0. (a) Phase 1 (b) Phase 2 (c) Phase 3 (d) Phase 4 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: HIQL: Phase-wise fANOVA main-effect importances under expert data quality for antmaze-large-v0. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: QRL: Phase-wise fANOVA main-effect importances under expert data quality for antmaze-large-v0. 10 6 10 5 10 4 10 3 10 2 Learning Rate 0.80 0.85 0.90 0.95 Discount Factor 1 2 3 4 10 6 10 5 10 4 10 3 10 2 Learning Rate 0.80 0.85 0.90 0.95 Discount Factor 1 2 3 4 (a) HIQL (Medium) (b) QRL (Medium) 10 6 10 5 10 4 10 3 10 2 Learning Rate 0.80 0.85 0.90 0.95 Discount Factor 1 2 3 4 10 6 10 5 10 4 10 3 10 2 Lear… view at source ↗
Figure 14
Figure 14. Figure 14: Pure exploration noise mobility plots. Top: Mobility results for the antmaze-medium-v0 environment. Bottom: Mobility results for the antmaze-large-v0 environment. Similar to the scheduled noise experiments, the representation learning method (QRL) maintains a more stable optimal hyperparameter region compared to the bootstrapped method (HIQL) even under exploration noise. 24 [PITH_FULL_IMAGE:figures/full… view at source ↗
Figure 15
Figure 15. Figure 15: 80% noise mobility plots. Visualization of hyperparameter landscape stability under high-noise conditions. Top: Mobility results for the antmaze-medium-v0 environment with 80% exploration noise. Bottom: Mobility results for the antmaze-large-v0 environment. These plots demonstrate how the near-optimal region shifts or remains anchored when the dataset is dominated by sub-optimal exploration transitions. 1… view at source ↗
Figure 16
Figure 16. Figure 16: 40% noise mobility plots. Hyperparameter sensitivity visualization for intermediate data quality levels. Top: Results for the antmaze-medium-v0 environment with 40% noise. Bottom: Results for the antmaze-large-v0 environment. At this intermediate noise level, we can observe the transition of the optimal hyperparameter region as the dataset becomes increasingly dominated by goal-directed navigation data. 2… view at source ↗
Figure 17
Figure 17. Figure 17: Pure expert data mobility plots. Visualization of the hyperparameter landscape using high-quality navigation data without exploration noise. Top: Results for the antmaze-medium-v0 environment. Bottom: Results for the antmaze-large-v0 environment. This setup represents the cleanest data regime, allowing for a direct comparison of how HIQL and QRL perform when the data quality is near-optimal. 10 6 10 5 10 … view at source ↗
Figure 18
Figure 18. Figure 18: Pure expert data across diverse environments. Evaluation of hyperparameter landscape stability on varying morphologies and dynamics. Top: Ant-Teleport environment. Middle: Humanoid-Medium environment. Bottom: PointMaze-Medium environment. Even across these diverse configurations, the representation-based objective (QRL) consistently demonstrates a more stable optimal region in the learning-rate – discount… view at source ↗
Figure 19
Figure 19. Figure 19: Mobility plots across data quality levels for the final training phase. We visualize how the optimal hyperparameter region drifts when evaluating the last phase of training across different data mixtures. Top: Results for antmaze-medium-v0. Bottom: Results for antmaze-large-v0. These plots isolate the learning mechanism’s sensitivity to the final data distribution, highlighting the contrast between the st… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 4 linked inside Pith

  1. [7]

    Kumar, A

    A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conser- vative q-learning for offline reinforcement learning. In Larochelle et al. (2020). H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H. Lin, editors.Proceedings of the 34th International Conference on Advances in Neural Information Process- ing Systems (NeurIPS’20),

    2020

  2. [9]

    URL https://arxiv.org/abs/2005.01643. M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and F. Hut- ter. SMAC3: A versatile bayesian optimization package for Hyperparameter Optimization.Journal of Machine Learning Research, 23(54):1–9,

    Pith/arXiv arXiv 2005

  3. [10]

    Y . Mao, H. Zhang, C. Chen, Y . Xu, and X. Ji. Supported value regularization for offline reinforcement learning. In Oh et al. (2023). A. Mohan, C. Benjamins, K. Wienecke, A. Dockhorn, and M. Lindauer. Autorl hyperparameter landscapes. In A. Faust, C. White, F. Hutter, R. Garnett, and J. Gardner, editors,Proceedings of the Second International Confer- enc...

    2023

  4. [11]

    Myers, C

    V . Myers, C. Ji, and B. Eysenbach. Horizon generalization in reinforcement learning. InThe Thirteenth International Conference on Learning Representations (ICLR’25)ICL (2025). Published online:iclr.cc. V . Myers, B. Chunyuan Zheng, B. Eysenbach, and S. Levine. Offline goal-conditioned reinforcement learning with quasimetric representations. InProceedings...

    Pith/arXiv arXiv 2025

  5. [14]

    R. Sass, E. Bergman, A. Biedenkapp, F. Hutter, and M. Lin- dauer. Deepcave: An interactive analysis tool for auto- mated machine learning. In M. Mutny, I. Bogunovic, W. Neiswanger, S. Ermon, Y . Yue, and A. Krause, ed- itors,ICML Adaptive Experimental Design and Active Learning in the Real World (ReALML Workshop 2022),

    2022

  6. [17]

    Zheng, V

    B. Zheng, V . Myers, B. Eysenbach, and S. Levine. Multi- step quasimetric learning for scalable goal-conditioned reinforcement learning.CoRR, abs/2511.07730,

    arXiv

  7. [18]

    15 B.2 Hierarchical Implicit Q-Learning (HIQL)

    12 When Are RL Hyperparameters Benign? Appendix Table of Contents A Related Work 13 B Algorithmic Details 15 B.1 Self-Supervised Goal Relabeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 B.2 Hierarchical Implicit Q-Learning (HIQL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 B.3 Quasimetric RL...

    2018

  8. [19]

    This differs from weight-space loss landscape analyses in deep networks, which study geometric properties of the training objective in parameter space

    for studying structure in thehyperparameter response surface(configuration → performance), including robustness, modality, and the geometry of near-optimal regions (Pitzer and Affenzeller, 2012; Malan, 2021). This differs from weight-space loss landscape analyses in deep networks, which study geometric properties of the training objective in parameter spa...

    2012

  9. [21]

    A recent line of work in this domain focuses explciitly on non-TD objectives (Wang et al., 2023; Myers et al., 2025a;b; Zheng et al.,

    that interpret parts of goal-reaching as near-supervised representation/behavior learning. A recent line of work in this domain focuses explciitly on non-TD objectives (Wang et al., 2023; Myers et al., 2025a;b; Zheng et al.,

    2023

  10. [22]

    that utilize the (quaismetric) structure of goal distances (Mohan et al., 2024). This structure is central to our study: it lets us vary data quality systematically while holding interaction budgets fixed, and it exposes how bootstrapping-based objectives (e.g., IQL/HIQL-style backups) can change the optimization problem even when exploration is not a con...

    2024

  11. [24]

    All configurations within a phase are evaluated from the same reference checkpoint to ensure comparability

    Inputs to fANOV A.For eachphase t,algorithm(QRL, HIQL), anddataset regime(constant mixture or scheduled data quality), we collect a set of evaluated configurations Ct ={(λ (i), ft(λ(i)))}nt i=1, where λ(i) is a hyperparameter configuration and ft(λ(i)) is the corresponding phase-indexed evaluation score produced by our phased protocol (Section 3). All con...

    2023

  12. [512]

    For details, refer to OGBench (Park et al., 2025a). D. Phased Training To gather data during training, we apply the phased pipeline proposed by Mohan et al. (2023), as shown in Figure 4, to offline GCRL. The resulting data will allow us to study the hyperparameter landscapes throughout the training process. First, we start training a set of configurations...

    2023

  13. [1996]

    T. Wang, A. Torralba, P. Isola, and A. Zhang. Optimal goal- reaching reinforcement learning via quasimetric learning. In Krause et al. (2023). R. Yang, Y . Lu, W. Li, H. Sun, M. Fang, Y . Du, X. Li, L. Han, and C. Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. InThe Tenth International Conference on Learning Repre...

    2023

  14. [2002]

    Fujimoto and S

    S. Fujimoto and S. Gu. A minimalist approach to offline reinforcement learning. In Ranzato et al. (2021). S. Fujimoto, D. Meger, and D. Precup. Off-policy deep rein- forcement learning without exploration. In K. Chaudhuri and R. Salakhutdinov, editors,Proceedings of the 36th In- ternational Conference on Machine Learning (ICML’19), volume

    2021

  15. [2008]

    Kostrikov, A

    I. Kostrikov, A. Nair, and S. Levine. Offline reinforce- ment learning with implicit q-learning. InThe Tenth International Conference on Learning Representations (ICLR’22)ICL (2022). Published online:iclr.cc. A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors.Proceedings of the 40th Interna- tional Conference on Machine Le...

    2022

  16. [2014]

    near-optimal

    via DeepCA VE (Segel et al., 2025). This yields marginal importance scores for individual hyperparameters as well as interaction mass capturing higher-order effects. To quantify changes in importance profiles across phases, we computed cosine distance between phase-wise importance vectors. F.5. Reproducibility details All experiments are conducted with fi...

    2025

  17. [2015]

    and hindsight relabeling (Andrychowicz et al., 2017), and connects to goal-conditioned supervised learning perspectives (Yang et al.,

    2017

  18. [2018]

    Tarasov, A

    D. Tarasov, A. Nikulin, D. Akimov, V . Kurenkov, and S. Kolesnikov. Corl: Research-oriented deep offline rein- forcement learning library. In Oh et al. (2023). J. Tsitsiklis and B. Van Roy. Analysis of temporal- diffference learning with function approximation

    2023

  19. [2019]

    Ghosh, C

    D. Ghosh, C. Bhateja, and S. Levine. Reinforcement learn- ing from passive data via latent intentions. In Krause et al. (2023). P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In S. McIlraith and K. Weinberger, editors,Proceedings of the Thirty-Second Conference on Artificial Intelligence (...

    2023

  20. [2020]

    Agarwal, M

    R. Agarwal, M. Schwarzer, P. Samuel Castro, A. C. Courville, and M. G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Ran- zato et al. (2021). M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. InProceed- ings of the 32nd ...

    2021

  21. [2021]

    Eimer, M

    T. Eimer, M. Lindauer, and R. Raileanu. Hyperparameters in reinforcement learning and how to tune them. In Krause et al. (2023). L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Implementation matters in deep RL: A case study on PPO and TRPO. InThe Eigth International Conference on Learning Representations (ICLR’20). ICLR,

    2023

  22. [2022]

    X. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage- weighted regression: Simple and scalable off-policy re- inforcement learning.arXiv preprint arXiv:1910.00177,

    Pith/arXiv arXiv 1910

  23. [2023]

    Curran Asso- ciates. S. Park, D. Ghosh, B. Eysenbach, and S. Levine. Hiql: Offline goal-conditioned rl with latent states as actions. In Oh et al. (2023). S. Park, K. Frans, B. Eysenbach, and S. Levine. Og- bench: Benchmarking offline goal-conditioned rl. In The Thirteenth International Conference on Learning Representations (ICLR’25)ICL (2025). Published...

    2023

  24. [2025]

    Published on- line:iclr.cc

    ICLR. Published on- line:iclr.cc. M. Jaderberg, V . Dalibard, S. Osindero, W. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, C. Fernando, and K. Kavukcuoglu. Popula- tion based training of neural networks.arXiv:1711.09846 [cs.LG],

    Pith/arXiv arXiv