Test-time Offline Reinforcement Learning on Goal-related Experience

Andreas Krause; Georg Martius; Jonas H\"ubotter; Marco Bagatella; Mert Albaba

arxiv: 2507.18809 · v2 · pith:YDNCYSZEnew · submitted 2025-07-24 · 💻 cs.LG

Test-time Offline Reinforcement Learning on Goal-related Experience

Marco Bagatella , Mert Albaba , Jonas H\"ubotter , Georg Martius , Andreas Krause This is my paper

Pith reviewed 2026-05-19 02:15 UTC · model grok-4.3

classification 💻 cs.LG

keywords test-time trainingoffline reinforcement learninggoal-conditioned policiesdata selectionpolicy fine-tuningloco-navigationmanipulation tasksreinforcement learning

0 comments

The pith

Selecting transitions relevant to the current state and test goal from an offline dataset, then fine-tuning the policy on them for a few gradient steps, produces better goal-conditioned performance than standard offline pre-training.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that offline goal-conditioned reinforcement learning benefits from a form of test-time adaptation. A self-supervised criterion picks out transitions that match the present state and perform well relative to the evaluation goal. Brief fine-tuning on this selected subset then improves the policy. The process repeats in a receding-horizon manner as the agent rolls out its trajectory. This yields clear gains on high-dimensional loco-navigation and manipulation tasks while using only modest extra compute.

Core claim

A self-supervised data selection criterion that ranks offline transitions by relevance to the current state and quality with respect to the evaluation goal enables a few gradient steps of fine-tuning to produce substantial performance improvements over a standard offline pre-trained goal-conditioned policy across a wide range of high-dimensional loco-navigation and manipulation tasks; the routine is applied in receding-horizon fashion during evaluation.

What carries the argument

The self-supervised data selection criterion that scores transitions according to relevance to the current state and quality with respect to the evaluation goal.

If this is right

Fine-tuning the policy on the selected subset for a few gradient steps leads to significant performance gains over standard offline pre-training.
The adaptation routine can be applied repeatedly during evaluation in a receding-horizon fashion to adjust the policy to the unfolding trajectory.
At comparable inference-time compute budgets, the method achieves gains that cannot be matched simply by increasing model size.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same selection-plus-fine-tuning pattern may extend usefully to non-goal-conditioned offline RL settings where a suitable relevance signal can be defined.
Lightweight test-time updates could be combined with other forms of policy adaptation such as prompt tuning or memory retrieval in broader agent architectures.
The approach invites experiments that vary the size and quality distribution of the offline dataset to test how robust the selection criterion remains.

Load-bearing premise

The selection criterion reliably identifies transitions that improve the policy without introducing bias or discarding information needed for further gains.

What would settle it

An experiment in which fine-tuning on the selected transitions produces no improvement or degrades performance relative to the untouched pre-trained policy on the same tasks would falsify the central claim.

Figures

Figures reproduced from arXiv: 2507.18809 by Andreas Krause, Georg Martius, Jonas H\"ubotter, Marco Bagatella, Mert Albaba.

**Figure 2.** Figure 2: GC-TTT specializes the agent to the next steps for achieving its target goal. the agent’s current state s ∈ S, we leverage a notion of temporal distance. In practice, this can be estimated by the learned quasimetric −V (s, g) of a value function estimate (Wang et al., 2023) or by the locally correct distance function d conventionally exposed by the goal-conditioned reward function (Andrychowicz et al., 201… view at source ↗

**Figure 3.** Figure 3: Visualization of data selection by GC-TTT in antmaze play during one evaluation episode (in orange). A random subset of trajectories from the dataset is shown in gray. While this operation selects sub-trajectories that are relevant to the agent’s current state, not all of them might be useful for reaching the agent’s target goal g ⋆ ∈ G. We thus further filter the data to include only those sub-trajectorie… view at source ↗

**Figure 4.** Figure 4: The four environments from OGBench (Park et al., 2025): from top left in clockwise order, humanoidmaze, cubesingle, antmaze, pointmaze. We evaluate all environments in their medium instance, across two datasets of different qualities, namely navigate and stitch. The former includes full demonstrations for any evaluation state-goal pair, while the latter may only be solved by “stitching” different trajector… view at source ↗

**Figure 5.** Figure 5: Success rates of GC-TTT within each environment, averaged across RL backbones. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

**Figure 6.** Figure 6: Left: Ablation of the data selection criteria. Both relevance and optimality have to be considered to filter the dataset for test-time training. Middle: Allocating more compute by increasing the frequency of TTT improves performance, and saturates slightly earlier in simpler environments. Right: We compare scaling test-time compute of GC-TTT (by increasing TTT frequency) to scaling the policy networks such… view at source ↗

read the original abstract

Foundation models compress a large amount of information in a single, large neural network, which can then be queried for individual tasks. There are strong parallels between this widespread framework and offline goal-conditioned reinforcement learning algorithms: a universal value function is trained on a large number of goals, and the policy is evaluated on a single goal in each test episode. Extensive research in foundation models has shown that performance can be substantially improved through test-time training, specializing the model to the current goal. We find similarly that test-time offline reinforcement learning on experience related to the test goal can lead to substantially better policies at modest compute costs. We propose a novel self-supervised data selection criterion, which selects transitions from an offline dataset according to their relevance to the current state and quality with respect to the evaluation goal. We demonstrate across a wide range of high-dimensional loco-navigation and manipulation tasks that fine-tuning a policy on the selected data for a few gradient steps leads to significant performance gains over standard offline pre-training. Our goal-conditioned test-time training (GC-TTT) algorithm applies this routine in a receding-horizon fashion during evaluation, adapting the policy to the current trajectory as it is being rolled out. Finally, we study compute allocation at inference, demonstrating that, at comparable costs, GC-TTT induces performance gains that are not achievable by scaling model size.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

GC-TTT shows that quick test-time fine-tuning on selected offline data can improve goal-conditioned policies more efficiently than model scaling, but the selection step risks biasing away from useful dynamics information.

read the letter

The main thing to know is that this paper demonstrates modest test-time adaptation on goal-related offline experience can produce clear gains over standard pre-training for goal-conditioned policies, and that the same compute budget spent on adaptation beats spending it on larger models instead. They apply the adaptation in a receding-horizon loop during evaluation, which fits naturally with how these policies are actually used in robotics tasks.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Goal-Conditioned Test-Time Training (GC-TTT) for offline goal-conditioned reinforcement learning. It proposes a self-supervised criterion to select relevant and high-quality transitions from an offline dataset for the current test goal, followed by a few gradient steps of policy fine-tuning. This process is repeated in a receding-horizon fashion during evaluation. The authors report substantial performance improvements over standard offline pre-training on various high-dimensional locomotion, navigation, and manipulation tasks, and show that these gains cannot be matched by simply scaling the model size at equivalent compute budgets.

Significance. If the reported gains are robust, this work highlights the potential of test-time adaptation in offline RL by specializing policies to specific goals using selected experience, analogous to test-time training in foundation models. It offers a compute-efficient way to improve policy performance at inference without retraining from scratch or increasing model capacity, which could be impactful for deploying goal-conditioned agents in diverse environments.

major comments (2)

[§4.2] §4.2 (Data Selection): The self-supervised criterion (relevance to current state + quality w.r.t. evaluation goal) is likely to favor high-return trajectories when quality is scored via existing value estimates or returns in the offline data. This risks excluding dynamics-critical or failure-mode transitions needed for robust goal-conditioned improvement in high-dimensional loco-navigation and manipulation. The central performance claim would be strengthened by an ablation comparing selected vs. random/full-dataset fine-tuning under identical gradient-step budgets.
[§5.3] §5.3 (Compute Allocation): The comparison to model scaling does not test whether equivalent compute spent on fine-tuning the original (non-selected) dataset yields comparable or better gains. Without this control, it remains unclear whether the selection step itself, rather than the fine-tuning procedure, is responsible for the reported improvements.

minor comments (2)

[Abstract] Abstract: The claim of 'significant performance gains' would be more informative if accompanied by a brief quantitative summary (e.g., average success-rate improvement and number of tasks).
[Method] Notation: The precise functional form of the relevance and quality scores (including any learned components or thresholds) should be stated explicitly in the method section to aid reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful and constructive feedback. We address each major comment below with clarifications and commitments to strengthen the manuscript.

read point-by-point responses

Referee: [§4.2] §4.2 (Data Selection): The self-supervised criterion (relevance to current state + quality w.r.t. evaluation goal) is likely to favor high-return trajectories when quality is scored via existing value estimates or returns in the offline data. This risks excluding dynamics-critical or failure-mode transitions needed for robust goal-conditioned improvement in high-dimensional loco-navigation and manipulation. The central performance claim would be strengthened by an ablation comparing selected vs. random/full-dataset fine-tuning under identical gradient-step budgets.

Authors: We acknowledge the referee's concern that the quality component of our selection criterion, which incorporates value estimates or returns, could bias toward high-return trajectories and potentially omit failure-mode or dynamics-critical transitions. Our criterion is self-supervised and combines state relevance with goal-directed quality, but we agree this does not explicitly guarantee inclusion of all failure cases. To strengthen the central claim, we will add an ablation in the revised manuscript comparing fine-tuning on selected data versus random sampling or the full dataset, using identical gradient-step budgets. Preliminary internal checks suggest the selection improves sample efficiency, but the new experiment will provide direct evidence. revision: yes
Referee: [§5.3] §5.3 (Compute Allocation): The comparison to model scaling does not test whether equivalent compute spent on fine-tuning the original (non-selected) dataset yields comparable or better gains. Without this control, it remains unclear whether the selection step itself, rather than the fine-tuning procedure, is responsible for the reported improvements.

Authors: We agree that our existing comparison to model scaling at matched compute does not fully isolate the contribution of the selection step versus fine-tuning on unselected data. The manuscript emphasizes that GC-TTT gains exceed those from capacity scaling, but an additional control with equivalent fine-tuning compute on the full non-selected dataset would clarify whether selection is essential. We will run and report this experiment in the revision to address this point directly. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical algorithm with independent performance claims

full rationale

The paper describes an algorithmic procedure (self-supervised data selection by relevance and quality, followed by short fine-tuning) and reports empirical gains on loco-navigation and manipulation tasks. No equations or derivations are presented that reduce the claimed improvement to a quantity defined by the selection criterion itself or by fitted parameters. The central result is a set of benchmark comparisons, not a closed-form prediction. Self-citations, if present, are not load-bearing for any uniqueness theorem or ansatz that would force the outcome. The method is therefore self-contained against external task evaluations.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

The central claim rests on the availability of sufficiently rich offline datasets containing goal-related transitions and on the effectiveness of short fine-tuning without catastrophic forgetting or instability.

free parameters (2)

number of fine-tuning gradient steps
Chosen to achieve adaptation at modest compute cost; value is not derived from first principles.
data selection thresholds or scoring weights
Parameters that determine relevance and quality cutoffs for transition selection.

axioms (1)

domain assumption The offline dataset contains transitions that are both relevant to test-time states and high-quality for the evaluation goal.
Invoked when claiming that self-supervised selection can produce useful fine-tuning data.

pith-pipeline@v0.9.0 · 5774 in / 1342 out tokens · 100703 ms · 2026-05-19T02:15:39.549708+00:00 · methodology

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Learning to Discover at Test Time
cs.LG 2026-01 unverdicted novelty 7.0

TTT-Discover applies test-time RL to set new state-of-the-art results on math inequalities, GPU kernels, algorithm contests, and single-cell denoising using an open model and public code.
Dejavu: Towards Experience Feedback Learning for Embodied Intelligence
cs.RO 2025-10 unverdicted novelty 6.0

Dejavu augments frozen VLA policies with an Experience Feedback Network that retrieves relevant past trajectories and uses RL-trained semantic similarity rewards to enable post-deployment adaptation in embodied tasks.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · cited by 2 Pith papers · 6 internal anchors

[1]

Bertolissi, J

Ryo Bertolissi, Jonas Hübotter, Ido Hakimi, and Andreas Krause. Local mixtures of experts: Essentially free test-time training via model merging. arXiv preprint arXiv:2505.14136,

work page arXiv
[2]

arXiv preprint arXiv:2410.24164,

work page internal anchor Pith review Pith/arXiv arXiv
[3]

One-minute video generation with test-time training

Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, et al. One-minute video generation with test-time training. arXiv preprint arXiv:2504.05298,

work page arXiv
[4]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

work page internal anchor Pith review Pith/arXiv arXiv
[5]

Discover: Automated curricula for sparse-reward reinforcement learning

Leander Diaz-Bone, Marco Bagatella, Jonas Hübotter, and Andreas Krause. Discover: Automated curricula for sparse-reward reinforcement learning. arXiv preprint arXiv:2505.19850,

work page arXiv
[6]

Dynamic Evaluation of Transformer Language Models

Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of trans- former language models. arXiv preprint arXiv:1904.08378,

work page internal anchor Pith review Pith/arXiv arXiv 1904
[7]

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177,

work page internal anchor Pith review Pith/arXiv arXiv 1910
[8]

Learning to (Learn at Test Time): RNNs with Expressive Hidden States

Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620,

work page internal anchor Pith review Pith/arXiv arXiv
[9]

Flattening hierarchies with policy bootstrapping

John L Zhou and Jonathan C Kao. Flattening hierarchies with policy bootstrapping. arXiv preprint arXiv:2505.14975,

work page arXiv
[10]

TTRL: Test-Time Reinforcement Learning

Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084,

work page internal anchor Pith review Pith/arXiv arXiv
[11]

Test-Time Training

12 A Taxonomy of Test-time Training Test-time training (TTT) describes a family of methods that update model parameters at test-time for each task. We categorize various approaches to TTT below. Category Methods Imitating expert data Often referred to as “Test-Time Training” (TTT), e.g., Hardt & Sun (2024); Hübotter et al. (2025); Akyürek et al. (2025) Le...

work page 2024
[12]

is an offline RL algorithm, which avoids querying the critic on out-of-distribution actions, and directly estimates a value function through expectile regression. Given a distribution µ of state-action-next state transitions labeled with a reward, IQL defines the following losses: LQ(ϕ) = E(s,a,r,s′)∼µ (r + γVψ(s′) − Qϕ(s, a))2, (10) and LV (ψ) = E(s,a,r)...

work page 2022
[13]

is an offline reinforcement learning algorithm designed to flatten hierarchi- cal approaches (Park et al., 2023). At its core, it relies on implicit Q-learning for estimating a value function, and on AWR for policy extraction, with an additional term encouraging alignment of the low-level policies across close and distant goals: LSAW(θ) = −E(s,a,r)∼µ exp ...

work page 2023
[14]

D.1 Hyperparameters GC-TTT introduces some additional hyperparameters

D Implementation Details For environments and backbone algorithms, we adopt the default hyperparameters presented in OGBench (Park et al., 2025). D.1 Hyperparameters GC-TTT introduces some additional hyperparameters. We keep the percentile fixed at q = 0.2 and tune the remaining ones, including the horizonK, the number of gradient steps N, and the fine-tu...

work page 2025

[1] [1]

Bertolissi, J

Ryo Bertolissi, Jonas Hübotter, Ido Hakimi, and Andreas Krause. Local mixtures of experts: Essentially free test-time training via model merging. arXiv preprint arXiv:2505.14136,

work page arXiv

[2] [2]

arXiv preprint arXiv:2410.24164,

work page internal anchor Pith review Pith/arXiv arXiv

[3] [3]

One-minute video generation with test-time training

Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, et al. One-minute video generation with test-time training. arXiv preprint arXiv:2504.05298,

work page arXiv

[4] [4]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

work page internal anchor Pith review Pith/arXiv arXiv

[5] [5]

Discover: Automated curricula for sparse-reward reinforcement learning

Leander Diaz-Bone, Marco Bagatella, Jonas Hübotter, and Andreas Krause. Discover: Automated curricula for sparse-reward reinforcement learning. arXiv preprint arXiv:2505.19850,

work page arXiv

[6] [6]

Dynamic Evaluation of Transformer Language Models

Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of trans- former language models. arXiv preprint arXiv:1904.08378,

work page internal anchor Pith review Pith/arXiv arXiv 1904

[7] [7]

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177,

work page internal anchor Pith review Pith/arXiv arXiv 1910

[8] [8]

Learning to (Learn at Test Time): RNNs with Expressive Hidden States

Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620,

work page internal anchor Pith review Pith/arXiv arXiv

[9] [9]

Flattening hierarchies with policy bootstrapping

John L Zhou and Jonathan C Kao. Flattening hierarchies with policy bootstrapping. arXiv preprint arXiv:2505.14975,

work page arXiv

[10] [10]

TTRL: Test-Time Reinforcement Learning

Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084,

work page internal anchor Pith review Pith/arXiv arXiv

[11] [11]

Test-Time Training

12 A Taxonomy of Test-time Training Test-time training (TTT) describes a family of methods that update model parameters at test-time for each task. We categorize various approaches to TTT below. Category Methods Imitating expert data Often referred to as “Test-Time Training” (TTT), e.g., Hardt & Sun (2024); Hübotter et al. (2025); Akyürek et al. (2025) Le...

work page 2024

[12] [12]

is an offline RL algorithm, which avoids querying the critic on out-of-distribution actions, and directly estimates a value function through expectile regression. Given a distribution µ of state-action-next state transitions labeled with a reward, IQL defines the following losses: LQ(ϕ) = E(s,a,r,s′)∼µ (r + γVψ(s′) − Qϕ(s, a))2, (10) and LV (ψ) = E(s,a,r)...

work page 2022

[13] [13]

is an offline reinforcement learning algorithm designed to flatten hierarchi- cal approaches (Park et al., 2023). At its core, it relies on implicit Q-learning for estimating a value function, and on AWR for policy extraction, with an additional term encouraging alignment of the low-level policies across close and distant goals: LSAW(θ) = −E(s,a,r)∼µ exp ...

work page 2023

[14] [14]

D.1 Hyperparameters GC-TTT introduces some additional hyperparameters

D Implementation Details For environments and backbone algorithms, we adopt the default hyperparameters presented in OGBench (Park et al., 2025). D.1 Hyperparameters GC-TTT introduces some additional hyperparameters. We keep the percentile fixed at q = 0.2 and tune the remaining ones, including the horizonK, the number of gradient steps N, and the fine-tu...

work page 2025