{"id":"1b6a26be-2879-4ea8-bebe-48b0f1d11418","arxiv_id":"2505.13144","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TempDATA augments offline datasets with latent-space transitions organized by temporal distance and outperforms prior offline model-based RL on goal-reaching benchmarks.","lead":"This paper introduces TempDATA, an offline reinforcement learning method that creates extra training transitions in a learned latent space organized by temporal distance, rather than in raw state space. The method reports strong results on long-horizon, sparse-reward robot tasks such as D4RL AntMaze and FrankaKitchen.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation isolates transition augmentation from distance-based reward shaping; the claimed mechanism is unverified.","rationale":"The reader's verdict is CONDITIONAL with the weakest assumption being out-of-support generalization of the learned distance and latent dynamics. I agree that this is a real risk, but I think the more immediate, testable gap is the absence of a control that isolates the augmentation from the intrinsic reward. The ablations in Figure 8 compare TempDATA (representation + intrinsic reward + latent rollouts) to baselines using sparse rewards, so the contribution of the rollout data is confounded. This is explicitly testable by disabling the rollout buffer. Additionally, the duplicated and inconsistent AntMaze tables (Section 5.1 vs the inserted 'Table 1.Caption' block) mean the quantitative basis of the central claim is currently unreliable; both issues should be resolved before the claim can be accepted. The theorem's gap between tau<1 in practice and the tau->1 limit in Theorem 4.2 is a further concern. My recommendation remains CONDITIONAL, with the added condition of running the no-rollout control and reconciling the two tables.","tokens_in":37365,"tokens_out":10473,"duration_ms":99947,"concrete_test":"Run TempDATA on AntMaze medium-play, medium-diverse, large-play, and large-diverse with the rollout buffer disabled (set sigma_bD=0 in Algorithm 1) while keeping the pretrained encoder, the latent dynamics model (untrained for rollouts), and the intrinsic reward from Eq. (6). Compare the average success rate over 8 seeds to full TempDATA and to a sparse-reward baseline. If the no-rollout variant matches full TempDATA, the augmentation is not the active ingredient and the central claim should be weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that augmenting transitions in a temporal-distance-aware latent space is what makes offline MBRL succeed on sparse-reward long-horizon tasks. But the reported gains are not isolated from the representation-based intrinsic reward. In the policy extraction phase (Sec. 4.3, Eq. 6), TempDATA trains with a dense potential-based reward r_tilde(s,s') = d(f(s'),g)-d(f(s),g), while the Section 5.5 ablation baselines ('vanilla RL', 'skill RL') are trained on the sparse goal reward from the relabeling procedure. The ablation thus contrasts (a) sparse reward without augmentation against (b) dense reward with augmentation; it cannot attribute the improvement to the augmented transitions. A control that keeps the same intrinsic reward but disables rollouts (sigma_bD=0 in Algorithm 1) is absent. Without it, the central mechanism is unverified: the gains could come entirely from the learned distance shaping, making the model-based augmentation decorative. The theoretical support (Theorem 4.2) does not resolve this because the practical expectile coefficients (0.95, 0.97, 0.7) are far from tau=1, and the proof in Appendix B confuses min and max in the Bellman target.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TempDATA, an offline model-based RL method that learns a temporal-distance-aware autoencoder, trains a latent dynamics model, generates augmented transitions in latent space, and then extracts a policy using offline RL with a potential-based intrinsic reward. The authors claim that TempDATA outperforms prior offline MBRL methods and matches or exceeds diffusion-based trajectory augmentation and goal-conditioned RL on D4RL AntMaze, FrankaKitchen, CALVIN, and pixel-based FrankaKitchen.","tokens_in":37656,"tokens_out":3612,"duration_ms":38728,"significance":"If the empirical claims are correct, TempDATA would be a notable advance: it would be the first offline MBRL method to succeed on sparse-reward, long-horizon goal-reaching benchmarks where prior model-based methods score near zero. The paper also ships a practical pipeline with code, and the external success-rate benchmark is independent of the learned representation. However, the current manuscript contains a data-reporting inconsistency in the central AntMaze table, and the ablation does not isolate transition augmentation from the dense intrinsic reward, so the main mechanistic claim is not yet established.","major_comments":[{"comment":"The central AntMaze results are reported inconsistently across the two copies of Table 1. For example, TempDATA's score on medium-play is 74.8±8.3 in the main table but 60.3±16.3 in the appendix version; large-diverse is 44.2±15.3 vs 55.2±15.3; ultra-diverse is 35.3±10.9 vs 62.2±8.0; and the total scores differ (520.2 vs 272.6). Because the headline claim rests on these numbers, the authors must reconcile the tables, clarify which numbers correspond to which experimental configuration, and provide the raw per-seed results.","section":"Table 1 (main text) vs Appendix Table 1 (page 8)"},{"comment":"The ablation does not isolate the effect of transition augmentation from the effect of the dense intrinsic reward. Baselines labeled 'vanilla RL' and 'skill RL' are trained with the sparse relabeled goal reward, while TempDATA uses the potential-based reward r̃(s,s') of Eq. (6) on all sampled data. A control that keeps Eq. (6) but disables model rollouts (σ_bD=0 in Algorithm 1) is missing, so the reported gains could come entirely from reward shaping rather than from augmented transitions. This is load-bearing for the paper's central claim and should be addressed with an additional ablation.","section":"Section 5.5, Figure 8; Eq. (6)"},{"comment":"The theorem and its proof are not aligned and do not support the practical configuration. The theorem states equality for τ=1, but the proof argues via a limit τ→1; the main text writes 'lim τ=1' in Eq. (3). The proof also asserts that dθ is 'bounded both above and below by the same shortest-path cost' and therefore 'converges exactly', which assumes the very equality being proved. Moreover, the practical expectile coefficients are 0.95, 0.97, and 0.7, far from 1, so the theorem does not justify the learned distances used in the method. Please provide a correct statement, a correct proof, or finite-τ guarantees, or explicitly position Theorem 4.2 as only an asymptotic motivation.","section":"Theorem 4.2 and Appendix B"},{"comment":"The Bellman target Bd is written with a max over θ, while the proof in Appendix B uses a min over actions. Although a min is appropriate for a distance/cost formulation, the inconsistency between the main text and the proof makes the derivation hard to verify. Please define the target precisely and state the action optimization consistently in both places.","section":"Section 4.1, Bellman target for Ltraj"}],"minor_comments":[{"comment":"The notation 'lim τ=1' should be 'lim_{τ→1}'.","section":"Eq. (3)"},{"comment":"Figure 8 reports only bar heights with no numerical values or error bars, and the 'boxed markers' are not defined in the caption; please add numerical results and a description of the marker convention.","section":"Section 5.5, Figure 8"},{"comment":"The text calls the heatmaps 'quantitative' but reports no quantitative metric; please either provide a quantitative evaluation of learned distances against true shortest paths or soften the wording.","section":"Section 5.6, Figure 9"},{"comment":"The phrase 'unlike prior MARL methods' appears to be a typo for 'MBRL methods'; please correct it.","section":"Section 5.1"},{"comment":"The sentence 'This work was supported by by Institute...' contains a duplicated 'by'; please fix.","section":"Acknowledgements"},{"comment":"There are several typos and formatting issues (e.g., 'pre-defiend' in Section 4.1, the garbled duplication of Table 1 in the appendix). A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The duplicate AntMaze table with substantially different numbers is the most serious issue in this submission; I would ask the authors to provide the full per-seed results and clarify which table is the correct one before the paper can be considered further. The missing reward-shaping control in the ablation is also central, since the paper's stated contribution is transition augmentation, not dense reward shaping. If the authors can reconcile the tables and add the missing control, the contribution could be salvageable, but both issues should be resolved in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Name],\n\nTwo things to know about arXiv:2505.13144. First, it reports a substantial empirical win for offline model-based RL on sparse-reward long-horizon tasks. TempDATA learns a temporal-distance-aware autoencoder, trains a latent dynamics model, and uses model rollouts to augment the offline dataset. On AntMaze, Kitchen, CALVIN, and pixel Kitchen, it beats standard MBRL baselines and roughly matches diffusion-based augmentation and goal-conditioned methods. Second, the paper's central mechanism is not actually isolated, and the PDF contains a duplicated table with inconsistent TempDATA numbers.\n\nWhat is new: the integration of a distance-expectile representation (IQL/HIQL style) with a latent dynamics model for offline augmentation. Individually these ingredients are known; the combination is new, and the evaluation is broad—including wall-clock time and aggregate RLiable plots. The design is modular, which is a real asset.\n\nThe soft spots, in order of severity.\n\n1. Table 1 appears twice with different TempDATA scores: medium-play is 74.8 in one version and 60.3 in the other; ultra-diverse is 35.3 vs 62.2. That is a red flag. It needs to be resolved before anything else.\n\n2. Theorem 4.2, the theoretical backing, does not hold up as written. The proof mixes a min with a max in the Bellman target, and the practical expectile coefficients (0.7–0.97) are far from τ=1. The theory does not cover the actual configuration.\n\n3. The ablation does not distinguish augmentation from reward shaping. TempDATA trains with the intrinsic distance reward of Eq. (6), while the baseline variants in the ablation appear to use the sparse goal reward. So the gains could come entirely from the dense reward, with the model-based augmentation contributing nothing. There is no control with the same intrinsic reward but with rollouts disabled. The paper's own Closing Statements ask how well the rollouts venture beyond the empirical support, but that is not the same as testing the mechanism.\n\nOn the positive side, the authors are honest about limitations and the empirical coverage is broader than most MBRL papers. The heatmap visualizations are a nice sanity check. But the core claim—that temporal-distance-aware transition augmentation causes the improvement—is, as of now, unverified.\n\nWho is this for: anyone working on offline goal-conditioned or model-based RL. It is worth serious refereeing, but it needs a revised version with a corrected table and a proper control. I would not cite it in its current form. I would send it back for major revision rather than desk reject it.\n\nBest.","headline":"TempDATA is a promising empirical contribution to offline MBRL for sparse-reward long-horizon tasks, but a duplicated table, a shaky theorem proof, and a missing ablation that isolates augmentation from reward shaping mean the core mechanism is not yet nailed down.","tokens_in":38148,"tokens_out":4164,"would_cite":false,"duration_ms":39624,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TempDATA shows that offline model-based reinforcement learning can solve sparse-reward, long-horizon goal-reaching tasks when transitions are augmented in a temporal-distance-aware latent space.","keywords":["offline reinforcement learning","model-based RL","temporal distance","latent representation","goal-conditioned RL","data augmentation","sparse reward","long-horizon tasks"],"falsifier":"Train TempDATA on an AntMaze-medium dataset from which the corridor connecting the start region to the goal region has been excised, then check whether the latent dynamics invent a path through the missing corridor (over-generalization) or the agent fails to reach the goal (distance learning confined to dataset-supported shortcuts). Alternatively, compute the rank correlation between $d(f(s),f(s_{\\text{goal}}))$ and true shortest-path length on states sampled from evaluation rollouts that never appeared in the training set; a low correlation would show the temporal metric does not generalize outside its support.","tokens_in":37177,"feed_emoji":"🧭","tokens_out":9581,"duration_ms":76833,"temperature":0.7,"pith_summary":"The paper claims that offline model-based reinforcement learning fails on sparse-reward, long-horizon goal-reaching tasks because it augments transitions in raw state space, where Euclidean closeness has little to do with how long a task actually takes, so synthetic rollouts cross obstacles or miss the goal. TempDATA instead learns a latent space in which the distance $d(f(s),f(s_{\\text{goal}}))$ between two encoded states approximates the shortest temporal path between them, enforced at both the trajectory level (how far a goal is) and the transition level (single-step coherence). A dynamics model trained in this latent space generates augmented transitions, and an intrinsic reward $\\tilde{r}(s,s') = d(f(s'),f(s_{\\text{goal}})) - d(f(s),f(s_{\\text{goal}}))$ supplies learning signal to any off-the-shelf offline RL algorithm. The paper reports that TempDATA outperforms prior offline model-based methods, often by large margins, and matches or surpasses diffusion-based trajectory augmentation and goal-conditioned RL on D4RL AntMaze, FrankaKitchen, CALVIN, and pixel-based FrankaKitchen. If correct, this means the temporal metric, rather than a better dynamics model, is the missing ingredient for offline model-based RL in long-horizon tasks.","feed_headline":"Latent temporal distance fixes offline model-based RL","feed_subtitle":"TempDATA augments transitions in a temporal metric space, beating prior MBRL on AntMaze, Kitchen, CALVIN.","key_machinery":"The temporal-distance-aware autoencoder is the load-bearing component. Its encoder $f: \\mathcal{S} \\to \\mathcal{Z}$ is trained with three objectives: state reconstruction; a trajectory-level expectile regression that drives $d(f(s),f(s_{\\text{goal}}))$ toward the shortest path / optimal value; and a transition-level constraint $d(f(s),f(s')) \\le d_0$ that keeps single steps temporally smooth. The latent dynamics model $\\zeta(z'|z,a)$ learns one-step transitions in this metric space, so that model rollouts respect the temporal geometry rather than cutting through walls or inventing shortcuts. The intrinsic reward $\\tilde r(s,s') = d(f(s'),f(s_{\\text{goal}})) - d(f(s),f(s_{\\text{goal}}))$ converts latent progress into a learning signal for any off-the-shelf offline RL algorithm; in practice the authors combine IQL-style expectile regression with an AWR-style weighted imitation objective, plus a skill variable selected at test time by pointing along the latent direction to the goal.","core_discovery":"TempDATA establishes that the central obstacle for offline model-based RL in long-horizon goal-reaching tasks is the space in which rollouts happen, not the quality of the learned dynamics. Its autoencoder is regularized so that the latent distance approximates the optimal goal-conditioned value, i.e., the shortest temporal path from $s$ to the goal (Theorem 4.2, in the limit as the expectile coefficient $\\tau \\to 1$), while a transition-level constraint keeps consecutive states temporally coherent. A latent forward model $\\zeta(z'|z,a)$ then generates augmented transitions entirely in this representation space, decodes them back to states, and the intrinsic reward $\\tilde r(s,s') = d(f(s'),f(s_{\\text{goal}})) - d(f(s),f(s_{\\text{goal}}))$ provides the learning signal. On D4RL AntMaze (including the Ultra levels where prior MBRL methods score zero), FrankaKitchen, CALVIN, and a pixel-based Kitchen, the augmented data lets a standard offline RL backbone exceed previous MBRL methods and compete with diffusion-based trajectory augmentation and hierarchical goal-conditioned RL.","pith_inferences":["Because the intrinsic reward is a difference of a learned potential (the latent distance), TempDATA's augmented rewards act as potential-based shaping; whenever the learned distance equals the true shortest-path value, the optimal policy is unchanged by the augmentation, which would explain why no reward model is needed.","The same latent distance could also support zero-shot planning in latent space (for example, model predictive control over $\\zeta$), rather than decoding every rollout back to raw states; the paper does not explore this.","A direct stress test would be to measure distance-preservation error on states visited during evaluation but absent from the training buffer; if the metric shatters on stochastic or partially observed variants of the same benchmarks, performance should collapse exactly in those regimes."],"forward_implications":["Prior model-based baselines that scored near zero on AntMaze (MOPO, COMBO, RAMBO, ROMI) can be lifted to competitive or better performance purely by changing where augmentation happens.","TempDATA matches or surpasses diffusion-based trajectory augmentation (GTA, SynthER, S4RL) without generative modeling of whole trajectories, and with markedly lower training time.","The learned temporal metric generalizes to arbitrary test-time goals, as shown by obstacle-aware distance heatmaps, so a single representation supports planning to unseen destinations.","The framework is modular: any offline RL algorithm can consume its augmented dataset, and the deterministic autoencoder can be replaced by a variational one for pixel-based inputs."],"supporting_citations":[{"why":"Supplies the quasimetric learning theory underlying the value-metric equivalence used in Proposition 4.1.","marker":"(Wang & Isola, 2022)"},{"why":"Establishes temporal distance as optimal goal-conditioned value, the equality the trajectory-level loss enforces.","marker":"(Wang et al., 2023)"},{"why":"Provides the expectile regression used to implement the Bellman target for latent distance and the IQL offline backbone.","marker":"(Kostrikov et al., 2022)"},{"why":"HIQL is the strongest goal-conditioned baseline and the codebase TempDATA extends with skill-conditioned training.","marker":"(Park et al., 2023a)"},{"why":"MOPO defines the model-based rollout protocol TempDATA inherits and is the primary paradigm it compares against.","marker":"(Yu et al., 2020)"},{"why":"ROMI, the reverse model-based imagination baseline, is the strongest prior offline MBRL method TempDATA benchmarks against.","marker":"(Wang et al., 2021)"},{"why":"D4RL supplies the AntMaze and Kitchen datasets used in every main evaluation.","marker":"(Fu et al., 2020)"},{"why":"GTA is the diffusion-based trajectory augmentation approach TempDATA matches or surpasses.","marker":"(Lee et al., 2024c)"},{"why":"Motivates the geometric autoencoder constraint R(θ) that keeps the latent space distance-preserving.","marker":"(Nazari et al., 2023)"},{"why":"Provides the test-time skill adaptation that selects the latent skill direction toward the goal at deployment.","marker":"(Park et al., 2024)"}],"fun_headline_variants":["Temporal distance in latent space boosts offline model-based RL","TempDATA: temporal latent distance improves offline MBRL","Latent temporal distance revives offline model-based RL","Offline MBRL fixed by temporal metric latent rollouts","Temporal distance in latent space: key to offline MBRL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The learned latent distance and the latent forward model must stay accurate for states and actions that lie outside the fixed dataset's support, because the augmented transitions that carry the policy to the goal are synthesized in that latent space — and the paper itself asks how well model-based rollouts venture beyond the empirical support.","fun_headline_variants_meta":{"raw":{"variants":["Temporal distance in latent space boosts offline model-based RL","TempDATA: temporal latent distance improves offline MBRL","Latent temporal distance revives offline model-based RL","Offline MBRL fixed by temporal metric latent rollouts","Temporal distance in latent space: key to offline MBRL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3030,"prompt_tokens":970,"completion_tokens":2060,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":1980}},"tokens_in":586,"tokens_out":2060,"duration_ms":16321,"temperature":1.0,"reasoning_tokens":1980,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:18:50.779626+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train TempDATA on an AntMaze-medium dataset from which the corridor connecting the start region to the goal region has been excised, then check whether the latent dynamics invent a path through the missing corridor (over-generalization) or the agent fails to reach the goal (distance learning confined to dataset-supported shortcuts). Alternatively, compute the rank correlation between $d(f(s),f(s_{\\text{goal}}))$ and true shortest-path length on states sampled from evaluation rollouts that never appeared in the training set; a low correlation would show the temporal metric does not generalize outside its support.","supporting_citations":[{"cited_title":"and Isola, P","cited_arxiv_id":null,"evidence_quote":"Supplies the quasimetric learning theory underlying the value-metric equivalence used in Proposition 4.1."},{"cited_title":"Optimal goal-reaching reinforcement learning via quasimetric learning","cited_arxiv_id":null,"evidence_quote":"Establishes temporal distance as optimal goal-conditioned value, the equality the trajectory-level loss enforces."},{"cited_title":"Offline reinforcement learning with implicit q -learning","cited_arxiv_id":null,"evidence_quote":"Provides the expectile regression used to implement the Bellman target for latent distance and the IQL offline backbone."},{"cited_title":"Offline reinforcement learning with reverse model-based imagination","cited_arxiv_id":null,"evidence_quote":"ROMI, the reverse model-based imagination baseline, is the strongest prior offline MBRL method TempDATA benchmarks against."},{"cited_title":"Geometric autoencoders--what you see is what you decode","cited_arxiv_id":null,"evidence_quote":"Motivates the geometric autoencoder constraint R(θ) that keeps the latent space distance-preserving."},{"cited_title":"Foundation policies with H ilbert representations","cited_arxiv_id":null,"evidence_quote":"Provides the test-time skill adaptation that selects the latent skill direction toward the goal at deployment."}],"review_version":1}