{"id":"4cd0f9d9-6667-4bde-ae5f-6ffdfd3b62e7","arxiv_id":"2602.05459","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In offline goal-conditioned RL, non-bootstrapped QRL shows broader and more stable hyperparameter tuning regions than bootstrapped HIQL, which drifts more as data quality changes.","lead":"This paper measures how much hyperparameter tuning offline goal-conditioned RL methods need, comparing a bootstrapped TD method (HIQL) with a non-bootstrapped quasimetric method (QRL). It finds QRL's good hyperparameter settings are broader and more stable across training, and links HIQL's brittleness to conflicting gradient signals across goals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Phase boundaries set from a single default config's 95% convergence may misalign HIQL and QRL training progress, making reported landscape differences protocol artifacts.","rationale":"The reader identified the phase-boundary placement as the weakest assumption. My analysis confirms this is the most load-bearing concern: it directly threatens the primary empirical observation. The paper's own limitation statement (D.1) attempts to preempt this criticism but does not address the asymmetric effect on cross-algorithm comparisons. A concrete test using absolute-step matching or a robust convergence criterion would settle whether the reported HIQL-vs-QRL differences persist. The causal attribution to bootstrapping is a secondary, acknowledged limitation, but the phase-boundary issue is more fundamental because it could invalidate the landscape evidence itself. The reader's CONDITIONAL verdict already captures this uncertainty, so no change to the verdict is needed. I agree with the reader's choice of weakest assumption.","tokens_in":25099,"tokens_out":7387,"duration_ms":80020,"concrete_test":"Recompute all phase-indexed landscape metrics (drift Δ, ε-optimality mass ρ_ε, early-selection regret, and fANOVA stability) with phase boundaries matched by absolute training steps (e.g., 10k, 25k, 50k, 100k) for both HIQL and QRL on the antmaze-medium and antmaze-large datasets, using the same configuration sets. Additionally, recompute the phases using a robust convergence criterion—e.g., the median of the 95%-convergence thresholds across all 128 configurations, or the best configuration's convergence—instead of a single default. If HIQL still shows significantly tighter optima and larger drift than QRL under step-matched phases, the phase-alignment concern is resolved. If the difference shrinks or reverses, the paper's central assertion requires revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparison of HIQL vs. QRL landscape geometry (drift, near-optimal mass, importance stability) rests on phase boundaries defined in Appendix D.1: each algorithm's phases are placed at fixed fractions of the last time a single hand-tuned default configuration crosses 95% of its final success rate (Table 10). For antmaze-medium-v0 at 0% explore, this threshold occurs at 18,461 steps for QRL but 99,734 steps for HIQL—a >5x difference. Consequently, the four phases for QRL span the absolute step range ~4.6k–18.5k, whereas HIQL spans ~25k–100k. QRL is thus evaluated almost entirely during the early, rapidly-improving portion of its training, while HIQL is evaluated across a much longer, later window. The paper's own D.1 acknowledges the reference config may be suboptimal and that phase placement can be inaccurate, but dismisses this as 'not problematic' because the goal is only to find interesting timesteps. That dismissal is not valid for cross-algorithm comparisons: if QRL's default converges quickly because it is comparatively well-tuned (or because the 95% bar is low for a weak final performance), then QRL's landscapes are sampled at a systematically earlier training stage than HIQL's. Early training tends to have broader, more forgiving optima; later training often exhibits sharper, drifting peaks. The reported differences in ε-optimality mass, drift, and gradient-interference correlations could therefore stem from comparing QRL at ~15% of its eventual training progress with HIQL at ~100%. This is a load-bearing threat to the central claim that bootstrapping (rather than phase-selection asymmetry) causes the observed divergence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The full text of the manuscript (title: \"When Are RL Hyperparameters Benign? A Study in Offline Goal-Conditioned RL\") studies whether hyperparameter sensitivity in offline goal-conditioned RL is intrinsic to the problem or amplified by the training objective. It compares the bootstrapped TD method HIQL with the non-bootstrapped quasimetric method QRL under controlled stationary and scheduled data-quality regimes. Using phase-resolved hyperparameter landscapes, fANOVA importance profiles, and an inter-goal gradient-alignment diagnostic, it reports that (i) offline GCRL landscapes are generally more benign than online RL landscapes, (ii) once modest expert data is present, QRL exhibits broad and stable near-optimal regions while HIQL shows sharper optima and larger phase-to-phase drift, and (iii) HIQL's critic gradients show a heavier negative tail of inter-goal cosine similarity, which co-occurs with the observed sensitivity. The paper frames the gradient result as a descriptive diagnostic rather than a proof of causality.","tokens_in":25452,"tokens_out":6193,"duration_ms":65497,"significance":"If the reported findings hold, they would be a useful contribution to AutoRL and offline GCRL: they suggest that the choice of learning objective, not just the data distribution or exploration, can shape the geometry of hyperparameter response surfaces and that non-bootstrapped objectives may be substantially easier to tune. The study is computationally extensive (128 configurations, 5 seeds per phase, multiple AntMaze variants), defines quantitative landscape metrics (ε-optimality mass, drift, early-selection regret), and includes explicit limitations. The gradient-interference diagnostic is an interesting mechanistic hypothesis that is clearly distinguished from an optimality criterion. However, the paper's central cross-algorithm comparison is built on only two algorithms that differ in many dimensions beyond bootstrapping, and the phase-alignment protocol introduces a potential confound (Appendix D.1, Table 10). The manuscript also contains a severe internal inconsistency: the abstract supplied at the top describes a different paper (four methods, AWR temperature, Cube/Scene) from the full text (two methods, learning rate/discount factor, AntMaze variants). As it stands, the su","major_comments":[{"comment":"The abstract states that the paper studies four methods (GCIQL, GCIVL, QRL, CRL) under a shared AWR extractor, with landscapes over learning rate and AWR temperature, evaluated on AntMaze, Cube, and Scene. The full text studies two methods (HIQL, QRL) over learning rate and discount factor, on AntMaze variants and related mazes, with no AWR temperature or Cube/Scene experiments. These are irreconcilably different studies. The reader cannot determine which claims are actually being made. This must be resolved before any further review.","section":"Abstract vs. Full Text"},{"comment":"Phase boundaries are defined from the last time a single hand-tuned default configuration crosses 95% of its final success rate. For antmaze-medium-v0 at 0% explore, this occurs at 18,461 steps for QRL and 99,734 steps for HIQL, a >5x difference. The four phases therefore cover very different absolute training windows for the two algorithms. The paper acknowledges this in D.1 but dismisses it as 'not problematic' because the goal is only to find interesting time steps. That dismissal is not valid for cross-algorithm comparisons of drift, near-optimal mass, and importance stability: if QRL's default is relatively well-tuned or its 95% bar is reached quickly, QRL's landscapes are sampled at systematically earlier absolute steps. The reported differences may be artifacts of the phase protocol rather than properties of the objectives. The authors should provide a sensitivity analysis with al","section":"Appendix D.1 / Table 10"},{"comment":"The headline quantitative differences are not statistically robust. For example, scheduled HIQL ρ0.9 = 0.12±0.10 vs. QRL 0.35±0.25 overlap within one standard deviation; across-phase change for HIQL 0.22±0.16 vs. QRL 0.17±0.11 also overlap. Moreover, HIQL and QRL differ in objective, architecture, regularization, and update rules, so the specific attribution to bootstrapping is not supported by the design. The limitations section correctly notes this, but the abstract and conclusion present the contrast as a finding. A concrete test would be a bootstrapped vs. non-bootstrapped variant of the same algorithm family, or an explicit statement that the paper only reports a two-point comparison.","section":"Tables 2–3 / Section 4.2"},{"comment":"fANOVA importances are computed from SMAC/Hypersweeper optimization traces, not from a uniform design (Appendix E). This makes the importance vectors design-dependent: SMAC's proposal distribution changes over time and differs by algorithm. The phase-to-phase cosine distances in Table 1 and the perplexity/stability plots in Figure 2 could therefore reflect differences in the optimization trajectory rather than intrinsic shifts in which hyperparameters matter. The authors should either recompute importances on the fixed Sobol configuration set or justify why the non-uniform design does not affect the cross-algorithm comparisons.","section":"Appendix E / Section 4.1"}],"minor_comments":[{"comment":"The full text contains multiple typos and inconsistencies in references/citations: 'Howeer' (Appendix A), 'Adavantage', 'quaismetric', 'explciitly', 'ICL' truncated citations, and incomplete venue entries. A careful proofread is needed.","section":"Throughout"},{"comment":"Table 4 reports phase-to-phase change aggregated over both algorithms, despite the text discussing algorithm-specific differences. Please split by algorithm or clarify that the table is a pooled summary.","section":"Table 4"},{"comment":"The gradient diagnostic κ(g,g') uses gradients with respect to the critic trunk. It would help to state explicitly how 'critic trunk' is defined for HIQL and QRL, and whether the comparison is on the same parameter dimensionality. The Pearson correlation of 0.43 in Section 4.3 should also report sample size and confidence interval.","section":"Section 3 / Eq. (7)–(8)"},{"comment":"The caption says 'top-10%' near-optimal region, while the text and Eq. (4) define ε-optimality mass with ε thresholds. Please use consistent terminology and indicate which ε is shown.","section":"Figure 1 / Caption"},{"comment":"The definition of 'last intersection with the performance threshold' should specify interpolation details and how ties are handled, since phase boundary placement is load-bearing for the conclusions.","section":"Appendix F.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to contain two different papers: the abstract describes a four-method study on Cube/Scene, while the full text describes a two-method study on AntMaze. This is not a case of a slightly mismatched abstract; it is a fundamental identity problem. The editor should ask the authors to confirm which study is being submitted and align the abstract, title, and body before sending the revised version back to reviewers. The technical concerns about phase-alignment and confounded algorithm comparison are substantial but potentially addressable through additional analyses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you actually get from this arXiv ID is a different paper than the abstract advertises. The metadata abstract describes GCIQL/GCIVL/CRL and extractability; the full text is “When Are RL Hyperparameters Benign?” comparing HIQL and QRL on trainability landscapes. If that is just a bad upload, fine, but it has to be fixed before any serious review.\n\nThe real paper is a decent empirical contribution. It transfers the phased hyperparameter-landscape protocol from Mohan et al. to offline GCRL, and adds an inter-goal gradient alignment diagnostic (cosine similarity between critic gradients for different relabeled goals). The finding—QRL’s near-optimal regions are broader and more stable than HIQL’s, and HIQL’s negative gradient-interference tail is heavier—is plausible and useful. The study is careful: metrics for ε-optimality mass, drift, regret, fANOVA importances; multiple environments; fixed configuration sets; and the limitations section explicitly says cross-algorithm comparisons do not establish causality. That is the right level of claim.\n\nSoft spots. The phase-boundary concern from the stress-test is real and load-bearing. Phase boundaries are set from a single default config’s 95% convergence time; Table 10 shows QRL hits that at 18k steps on antmaze-medium 0% explore vs HIQL at 99k. So the four “phases” for QRL cover early training, while HIQL covers a much later window. The paper’s dismissal in D.1—that these inaccuracies are “not problematic”—is too quick, because the geometry and drift metrics are exactly what could change with training stage. This is not fatal given the paper’s framing, but the claim that bootstrapping causes the divergence needs either matched phase schedules or a sensitivity analysis.\n\nSecond, HIQL and QRL differ in objective, architecture, regularization, and update rules. The paper says so, but that means the title question—“when are RL hyperparameters benign?”—is not answered generally; it is a difference between two particular algorithms.\n\nThird, the key numbers in Table 2 have overlapping standard deviations. That is acceptable for a hypothesis-generating study, but not a verification.\n\nFourth, no code is released yet.\n\nWho this is for: people working on AutoRL, offline GCRL evaluation, and hyperparameter sensitivity. A serious referee should see this after the abstract is corrected and the phase artifact is either addressed or shown to be orthogonal. I would accept it for review now, with a note that the mismatch must be resolved first.","headline":"A careful, honestly hedged empirical study of hyperparameter landscape geometry in offline GCRL; worth refereeing once the arXiv abstract is corrected and the phase-boundary artifact is addressed.","tokens_in":25942,"tokens_out":3302,"would_cite":true,"duration_ms":37008,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hyperparameter brittleness in offline goal-conditioned RL is largely a consequence of bootstrapped value objectives, not an unavoidable property of RL.","keywords":["offline reinforcement learning","goal-conditioned RL","hyperparameter sensitivity","hyperparameter landscapes","TD bootstrapping","quasimetric representation learning","gradient interference","phase-resolved analysis"],"falsifier":"On a fixed offline dataset, take a quasimetric objective and add a bootstrapped target term to its critic loss: if the inter-goal gradient cosine-similarity distribution gains a heavier negative tail and the hyperparameter landscape becomes sharper and more phase-sensitive (lower ε-optimality mass, higher phase-to-phase drift), the paper's causal story is supported; if those metrics stay unchanged, the attribution of brittleness to bootstrapping is falsified.","tokens_in":24974,"feed_emoji":"🎯","tokens_out":6230,"duration_ms":60346,"temperature":0.7,"pith_summary":"The paper asks whether hyperparameter sensitivity in deep RL is intrinsic or amplified by particular training mechanisms, and uses offline goal-conditioned RL to separate the two. It shows that, once modest expert data is present, a non-bootstrapped quasimetric objective produces broad, stable near-optimal hyperparameter regions, while a bootstrapped TD objective yields sharp optima that drift across training phases. The authors attribute the divergence to inter-goal gradient interference: bootstrapped targets make critic updates from different goal relabelings conflict more strongly, and the heaviness of that conflict correlates with landscape brittleness. The implication is that sensitivity can be designed away by choosing objectives that avoid recursive target coupling, reducing the need for phase-specific tuning schedules.","feed_headline":"Bootstrapped value targets make offline RL tuning brittle","feed_subtitle":"Non-bootstrapped quasimetric methods stay stable across training phases; a gradient-alignment diagnostic explains why.","key_machinery":"The central instrument is the phased hyperparameter-landscape analysis paired with an inter-goal gradient-alignment diagnostic. Landscapes are built by training a fixed set of hyperparameter configurations from shared checkpoints, evaluating at phase boundaries, and summarizing geometry with ε-optimality mass (the fraction of configurations within, say, 90% of the best), phase-to-phase drift (mean normalized performance change), and early-selection regret (loss from committing to a configuration chosen in an early phase). The mechanistic diagnostic computes, for a single minibatch and multiple goal relabelings, the cosine similarity between critic-loss gradients; negative values indicate des","core_discovery":"In offline goal-conditioned RL with fixed datasets and controlled data quality, the choice of value-learning objective changes not just peak success but the entire geometry and transferability of the hyperparameter performance landscape. QRL, which learns a quasimetric distance-to-goal without bootstrapping, keeps a large fraction of configurations near-optimal and keeps its important hyperparameters stable across phases once about 20% expert data is present. HIQL, which learns a value function via bootstrapped TD expectile regression, shows tighter optima, stronger interactions between learning rate and discount factor, and larger phase-to-phase drift, especially under scheduled improvement","pith_inferences":["If the gradient-interference story generalizes, any relabeling-based TD learner (including hindsight experience replay in online settings) should show the same phase-dependent brittleness; measuring cosine similarity between gradient updates from different relabelings in those settings is a direct test.","The comparison is observational, but it suggests a verifiable causal claim: adding a bootstrapped target term to QRL's critic loss should shift its gradient-alignment distribution toward a heavier negative tail and sharpen its landscape, while removing the bootstrap term from HIQL should do the reverse.","The phase-boundary calibration could be checked by recomputing drift and ε-optimality mass with boundaries aligned at equal training steps or equal performance progress; if the HIQL-versus-QRL gap shrinks, part of the reported effect is an artifact of the protocol.","The gradient-alignment diagnostic could be turned into an online monitor: track the negative-tail mass during training and trigger a learning-rate adjustment when it grows, effectively replacing hand-tuned phase schedules with a signal-driven schedule."],"forward_implications":["Offline goal-conditioned RL hyperparameter landscapes are generally more benign than online RL landscapes: even with degraded data, large fractions of configurations stay near-optimal, and most landscape reorganization happens early in training.","With at least modest expert data, non-bootstrapped quasimetric objectives can make narrow, phase-specific hyperparameter schedules unnecessary, since near-optimal regions stay broad and stable.","Bootstrapped TD methods require more careful and stage-dependent tuning, and early good configurations can become suboptimal under scheduled data-quality shifts.","Inter-goal gradient-alignment measurements offer a low-cost proxy for predicting which objectives or configurations will be brittle: a heavier negative tail in critic gradient cosine similarity co-occurs with lower performance and sharper optima.","Objective design could target reducing cross-goal gradient conflicts while retaining bootstrapping's sample-efficiency benefits."],"fun_headline_variants":["Success rates hide brittle tuning in offline goal-conditioned RL","Peak success alone doesn't reveal extractable behavior","Brittle optima vs broad basins in offline GCRL","Non-bootstrapped QRL stays stable across hyperparameters","Bootstrapped value targets make offline RL tuning brittle"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The phase boundaries used to define training stages are derived from when a single hand-tuned default configuration first reaches 95% of its final success rate, and since QRL and HIQL converge at very different speeds, the phases may compare the two algorithms at different effective training progress — so the reported drift differences could be partly an artifact of boundary placement.","fun_headline_variants_meta":{"raw":{"variants":["Success rates hide brittle tuning in offline goal-conditioned RL","Peak success alone doesn't reveal extractable behavior","Brittle optima vs broad basins in offline GCRL","Non-bootstrapped QRL stays stable across hyperparameters","Bootstrapped value targets make offline RL tuning brittle"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":2850,"prompt_tokens":808,"completion_tokens":2042,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":1960}},"tokens_in":552,"tokens_out":2042,"duration_ms":13837,"temperature":1.0,"reasoning_tokens":1960,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:12:42.276313+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed offline dataset, take a quasimetric objective and add a bootstrapped target term to its critic loss: if the inter-goal gradient cosine-similarity distribution gains a heavier negative tail and the hyperparameter landscape becomes sharper and more phase-sensitive (lower ε-optimality mass, higher phase-to-phase drift), the paper's causal story is supported; if those metrics stay unchanged, the attribution of brittleness to bootstrapping is falsified.","supporting_citations":[],"review_version":1}