Pith. sign in

REVIEW 3 major objections 4 minor 59 references

MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that filtering augmented goals through a locally Lipschitz-bounded dynamics model gives offline goal-conditioned supervised learning agents the ability to stitch separate trajectories, and that the resulting goal…

desk verdict A useful empirical augmentation method whose theoretical guarantee does not hold as stated. read the letter →

arxiv 2412.11410 v2 pith:T6ZZKFP2 submitted 2024-12-16 cs.LG

Xing Lei , Xuetao Zhang , Donglin Wang This is my paper · ORCID
classification cs.LG
keywords offlinegoal-conditionedreinforcementlearningweightedsupervisedgoaldataaugmentationtrajectorystitchinglocalLipschitzcontinuitylearneddynamicsmodelreachabilitymazenavigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline goal-conditioned reinforcement learning methods that imitate relabeled trajectories are simple and stable, but they cannot stitch together fragments of different trajectories to reach goals that no single trajectory achieves. This paper argues that goal data augmentation can fix that, provided the augmented goals are diverse, keep the original action optimal, and are actually reachable. The proposed method, MGDA, learns a one-step dynamics model under a local Lipschitz continuity penalty, uses it to find states near the original goal, and samples new goals from later in those states' trajectories. The paper proves that, under a distribution-smoothness assumption, this sampling matches the one-step stitching distribution up to an error bounded by the cluster radius times the Lipschitz constant, and shows empirically that adding MGDA improves four GCWSL methods on state-based and vision-based maze tasks.

What carries the argument

The load-bearing object is a locally Lipschitz-bounded one-step residual dynamics model $\hat{f}$, trained with a state-dependent slack-weighted prediction loss plus weight clipping to keep the model predominantly $L$-Lipschitz (Equation 4). Given the original goal $g$, the model predicts transitions from candidate states, and a candidate is declared a nearby state only when the predicted transition lands within distance $\delta$ of $g$; an augmented goal is then sampled from the suffix of the trajectory containing that nearby state. This mechanism implements Goal Reachability and Action Optimality, Theorem 1 bounds the model prediction error by $\epsilon + (K + \Delta(\lambda_n))\|s_n - g\|$, and Theorem 2 uses Assumption 1 to convert the filtered sampling into the one-step stitching distribution up to $O(\epsilon_k L_1)$.

What would settle it

Take a maze whose k-means clusters are cut by an impassable wall, so two states in one cluster have near-zero probability of reaching each other in one step, run MGDA on a GCWSL method, and measure the fraction of augmented goals that lie on the far side of the wall: if that fraction is large and the success rate drops to the SGDA/TGDA level, the claim fails in that regime. Alternatively, estimate $p_{\beta_h}(g \mid s,a)$ for same-cluster state pairs and check whether the Lipschitz inequality $|p_{\beta_h}(g \mid s,a) - p_{\beta_h}(s_n \mid s,a)| \le L_1\|g-s_n\|$ holds; a cluster where the empirical occupancy difference is large relative to distance refutes Assumption 1.

Watch

Extended reading notes

Core claim

The central claim is that MGDA's augmented-goal distribution is within $O(\epsilon_k L_1)$ of the one-step goal-reaching stitching distribution, written as $p_{\text{MGDA}}(g \mid s,a) = p_{\text{1-step}}(g \mid s,a) \pm O(\epsilon_k L_1)$, where $\epsilon_k$ is the largest intra-cluster distance after k-means and $L_1$ is the Lipschitz constant of the discounted state occupancy of the data-collecting policy. The theorem holds when Assumption 1 (distribution smoothness) and the local Lipschitz dynamics constraint are satisfied. In practice, the nearby-state search uses the learned model as a filter: a candidate state $u$ is accepted as a nearby state only if $\|g - u - \hat{f}(s_t, a_t)\| < \delta$, meaning the predicted one-step transition from the candidate reaches the goal, and the augmented goal is then drawn from the later portion of the trajectory containing $u$. The empirical claim is that this filtering selects goals that are reachable and action-compatible, improving the stitching ability of GCWSL methods beyond the SGDA and TGDA baselines.

Load-bearing premise

The load-bearing premise is Assumption 1: for every data-collecting policy, the discounted state occupancy is Lipschitz continuous in nearby states within a k-means cluster; if reachable and unreachable goals in the same cluster have very different occupancy probabilities, the $O(\epsilon_k L_1)$ error in Theorem 2 is not small and MGDA's augmented goals are not provably one-step-stitching goals.

Editorial extensions

If this is right

  • GCWSL methods gain stitching capability without dynamic programming: MGDA samples previously unseen (state, goal) combinations directly from the offline dataset and trains the policy on them.
  • More offline data alone does not unlock stitching: the paper's scaling experiments show DWSL does not improve when the dataset grows from $10^5$ to $10^8$ transitions, while adding MGDA does.
  • The three stated principles, goal diversity, action optimality, and goal reachability, characterize which augmented goals are usable; MGDA satisfies all three, whereas SGDA and TGDA each violate at least one.
  • The local Lipschitz penalty is load-bearing: the ablation study shows that replacing the penalized objective with plain MSE generally lowers success rates across GCWSL variants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $O(\epsilon_k L_1)$ error term points to a direct improvement not explored in the paper: reachability-aware clustering, or shrinking k-means cluster diameters near obstacles, should tighten the approximation and reduce the unreachable-goal failures the authors observe in harder vision tasks.
  • Because the guarantee is stated for one step of stitching, a natural extension is iterating MGDA to compose several one-step reachability filters into multi-step stitched paths, which the paper does not test.
  • The three principles and the dynamics-filter idea could transfer beyond GCWSL to other goal-conditioned supervised learners, such as sequence models, by replacing hindsight relabeling with MGDA-filtered relabeling.
  • A simpler alternative suggested by the method's own filter is to learn a one-step reachability classifier (reach versus not reach within $\delta$) instead of a continuous dynamics model, which would avoid compounding prediction error while preserving the same filtering decision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Model-based Goal Data Augmentation (MGDA), a goal augmentation method for offline Goal-Conditioned Weighted Supervised Learning (GCWSL). MGDA learns a dynamics model with a local Lipschitz regularization term, uses it to identify states that can reach an original goal in one step, and samples augmented goals from later portions of the trajectories through those nearby states. The authors introduce three principles for goal augmentation (diversity, action optimality, reachability) and claim that MGDA satisfies them while prior methods (SGDA, TGDA) do not. The main theoretical claim, Theorem 2, states that MGDA's augmented-goal distribution approximates the one-step stitching distribution up to O(ε_k L1). The empirical evaluation uses offline point-maze environments (Umaze, Medium, Large) in state-based and vision-based versions, comparing four GCWSL baselines (WGCSL, GoFar, DWSL, SMORE) with and without MGDA, and includes an ablation of the local Lipschitz assumption and a dataset-size study.

Significance. The paper addresses a real and recognized limitation of GCWSL methods, namely the lack of trajectory stitching. The empirical study is systematic: it covers four GCWSL baselines, two observation modalities, multiple maze difficulties, bootstrap confidence intervals, an LLC ablation, and a dataset-size comparison. The ablation showing that the local Lipschitz regularized dynamics model generally outperforms a plain MSE model is a useful mechanistic check. If the theoretical guarantee were made rigorous, the contribution would be a principled, model-based augmentation method with a clear niche among prior augmentation techniques. However, the current proof of Theorem 2 is incomplete in a load-bearing way, and there is a mismatch between the algorithm as printed and the algorithm as described in the text, so the paper is not yet ready for acceptance.

major comments (3)
  1. [Supplementary, Proof of Theorem 2, Eq. (13)] The proof of Theorem 2 silently drops the integral over unreachable states s'_n with the annotation "select states correspond to reachable goals by \hat f". This is an unstated oracle assumption: there is no bound in the paper on the probability that the learned dynamics model \hat f, trained with Eq. (4) and thresholded at \delta, misclassifies a geometrically close but dynamically unreachable state as reachable. Assumption 1 (Eqs. (6)-(7)) bounds occupancy differences for reachable and unreachable states separately, but it says nothing about the classification error of \hat f. Consequently, the claimed O(ε_k L1) bound in Eq. (8) does not follow from the stated assumptions alone. The theorem is only valid under an additional assumption, such as a bounded misclassification probability or a perfect reachability filter, neither of which is stated or proven.
  2. [Algorithm 1, line 10] The printed algorithm uses the condition ∥g − u − \hat f(s_t, a_t)∥ < δ, i.e., it evaluates the dynamics model at the sampled (s_t, a_t) pair, whereas the text and Figure 2 define a nearby state s_n as one for which the dynamics model makes the one-step transition reach the goal, which requires evaluating \hat f at (u, a_u) for some action a_u available at u. As written, line 10 does not check whether u reaches g; it checks whether g is close to the one-step residual of the original (s_t, a_t). This discrepancy means the implemented algorithm may not be the one analyzed in Theorem 2, and it also undermines the stated goal-reachability property. The algorithm should be corrected or the notation clarified so that the reachability check matches the definition of nearby state.
  3. [Supplementary, Proof of Theorem 1, assumptions (10)-(11)] The assumptions used in the proof of Theorem 1 are not cleanly stated. Inequality (10) writes ∥f(s_n, a) − f(g, a)∥ ≤ K∥s_n − g∥ and (11) writes a similar expression for \hat f, but f and \hat f are defined on S × A, while g is an element of the goal space G; the notation f(g, a) requires a state-goal mapping (e.g., g = φ(s)) and a corresponding state representation. Moreover, in (11) the left-hand side mixes the true model f and the learned model \hat f, which appears to be a typo: the intended statement should compare \hat f(s_n, a) and \hat f(g, a). These issues make the proof of Theorem 1 difficult to verify as written and should be fixed by defining the domain of f and \hat f over states (possibly identifying goals with states through φ) and correcting the Lipschitz statements.
minor comments (4)
  1. [Abstract and Introduction] There are several typos and grammatical errors, e.g., "researches has revealed", "M odel-based G oal D ata A ugmentation", and "in their ability to enhancing stitching capabilities". The paper should be carefully proofread.
  2. [Table 1] The symbols in Table 1 are inconsistent and hard to read: the SGDA row uses '%' where checkmarks or crosses are presumably intended. Please use a consistent notation such as ✓/✗ or filled/empty circles.
  3. [Algorithm 1, lines 7-10] The notation on line 8 is confusing: "k = dt+" appears to be a typo, and it is unclear what "CLUSTER(sl)" returns (a cluster index or a cluster centroid). Please rewrite lines 7-10 with clear variable names and types.
  4. [Experiments, Table 4] In Table 4, the hyperparameter named "K" is set to 2.0, but the paper does not explain how this value relates to the Lipschitz constant K in Theorem 1 or whether it is a fixed bound for the spectral normalization in Eq. (4). A brief explanation would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical evaluation is against external baselines, and the theoretical guarantee is explicitly borrowed from non-overlapping prior work rather than from a self-citation chain.

full rationale

Walking the derivation chain, MGDA's empirical claims are evaluated on the point-maze datasets from Ghugare et al. (2024) against SGDA, TGDA, and four GCWSL baselines; these comparisons are external and do not reduce to the paper's assumptions. The central theoretical guarantee, Theorem 2 (Eq. 8), is explicitly presented as 'Proof of Theorem 2 (Rephrased from Lemma D.2 of (Ghugare et al. 2024))', and Assumption 1 is attributed to the same non-overlapping prior work, so this is an external borrowing rather than a self-citation chain. Theorem 1's bound is conditional on the local Lipschitz property that Eq. 4 enforces by spectral normalization; it is a consequence of the training design, not an independent prediction used to justify the training objective, so it is not a fitted input renamed as a prediction. There is a genuine rigor gap in the supplement: after replacing the cluster term via Eqs. 6-7, the unreachable-state integral is struck out with only the annotation 'select states correspond to reachable goals by \hat f', and the proof does not bound the model's reachability-filter error. This means Theorem 2 as stated is not fully proven, but an unproven step is not a circular reduction: no equation is defined in terms of the target result, no fitted parameter is renamed as a prediction, and no load-bearing conclusion is justified only by the authors' own prior work. Accordingly, no circular steps are identified.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central empirical result rests on the offline maze benchmarks and the filtering quality of the learned dynamics model. The theory adds two assumptions on top of standard supervised learning: local Lipschitz dynamics, which the training objective explicitly enforces, and a smooth-occupancy condition inherited from TGDA. Several hyperparameters (lambda, K, delta, number of clusters, augmentation probability) are chosen by hand and materially affect the algorithm. No new physical or mathematical entities are introduced.

free parameters (6)
  • lambda (global Lipschitz regularization) = 1.0
    Controls the penalty strength for enforcing local Lipschitz continuity in Eq. 4; chosen by hand (Table 4).
  • K (Lipschitz bound for true dynamics) = 2.0
    Used in Theorem 1 and set as a hyperparameter in Table 4; not estimated from data.
  • delta (goal reach threshold) = 0.5
    Threshold for considering a goal reached in Eq. 2 and for the model-based reachability check in Algorithm 1.
  • lambda_n (state-dependent slack) = learned
    Per-data-point slack variable in Eq. 4 that adaptively weights the continuity penalty; not a scientific constant but a tunable network output.
  • C (k-means cluster count) = 20 (Umaze), 40 (Medium), 80 (Large)
    Number of clusters in the k-means pre-processing step, taken from TGDA and dataset-dependent.
  • epsilon (augmentation probability) = 0.5
    Probability of applying goal augmentation during training; shared with SGDA and TGDA in the comparison.
assumptions (5)
  • domain assumption Ground-truth dynamics f is locally K-Lipschitz at nearby states around the goal (Eq. 10 in supplementary).
    Used in the proof of Theorem 1 to translate state distance into model-error bound; typical for physical environments but not guaranteed for arbitrary offline data.
  • ad hoc to paper Learned dynamics f_hat is locally Delta(lambda_n)-Lipschitz and the spectral normalization in Eq. 4 enforces this.
    The paper enforces this bound through a loss term and weight normalization; the theorem then assumes what the training objective is designed to produce.
  • domain assumption Assumption 1: discounted state occupancy of the data-collecting policies is Lipschitz continuous with respect to nearby states (Eq. 6 and 7).
    Borrowed from Ghugare et al. 2024; needed for Theorem 2 and not verified on the maze datasets.
  • domain assumption The offline dataset is generated by a set of policies beta(a|s,h) with context h sampled from p(h).
    Standard in the stitching literature; used in Theorem 2's derivation.
  • domain assumption Goal space and state space share a metric, and the relabeled goal g = phi(s_i) is a deterministic mapping from states.
    Assumed in Eq. 2 and throughout; needed for the one-step reachability test to make sense.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learning." pith.science (2026). https://pith.science/paper/T6ZZKFP2

@misc{pith2026241211410,
  author       = {Pith},
  title        = {Pith review of: MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T6ZZKFP2}},
  note         = {Machine review of arXiv:2412.11410}
}
read the original abstract

Recently, a state-of-the-art family of algorithms, known as Goal-Conditioned Weighted Supervised Learning (GCWSL) methods, has been introduced to tackle challenges in offline goal-conditioned reinforcement learning (RL). GCWSL optimizes a lower bound of the goal-conditioned RL objective and has demonstrated outstanding performance across diverse goal-reaching tasks, providing a simple, effective, and stable solution. However, prior research has identified a critical limitation of GCWSL: the lack of trajectory stitching capabilities. To address this, goal data augmentation strategies have been proposed to enhance these methods. Nevertheless, existing techniques often struggle to sample suitable augmented goals for GCWSL effectively. In this paper, we establish unified principles for goal data augmentation, focusing on goal diversity, action optimality, and goal reachability. Based on these principles, we propose a Model-based Goal Data Augmentation (MGDA) approach, which leverages a learned dynamics model to sample more suitable augmented goals. MGDA uniquely incorporates the local Lipschitz continuity assumption within the learned model to mitigate the impact of compounding errors. Empirical results show that MGDA significantly enhances the performance of GCWSL methods on both state-based and vision-based maze datasets, surpassing previous goal data augmentation techniques in improving stitching capabilities.

Figures

Figures reproduced from arXiv: 2412.11410 by the authors.

Figure 1
Figure 1. Counter examples of generalized principles and re [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. (left) State searched by other goal data augmentation methods. (right) State searched by our MGDA, constrained by the dynamics model in its relationship to the goal. MGDA ensures that the searched state and goal g satisfy the one-step transition crite￾rion, thereby defining this searched state as a nearby state sn. the reliability of the learned model can be assured. We iden￾tify the nearby state under the metric de… view at source ↗
Figure 3
Figure 3. Performance of the original GCWSL methods and the impact of incorporating different goal augmentation approaches [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Performance comparison between the original GCWSL approach and its enhancement with MGDA on vision-based [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Ablation study on the local Lipschitz continuity (LLC) assumption. The results clearly show that the modified MSE [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Performance of GCWSL methods trained on three different offline dataset sizes, averaged across all point mazes. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 24 canonical work pages

  1. [1]

    Akakzia, A.; Colas, C.; Oudeyer, P.-Y.; Chetouani, M.; and Sigaud, O. 2020. Grounding language to autonomously-acquired skills via goal generation. arXiv preprint arXiv:2006.07185

    arXiv 2020

  2. [2]

    Andrychowicz, M.; Wolski, F.; Ray, A.; Schneider, J.; Fong, R.; Welinder, P.; McGrew, B.; Tobin, J.; Pieter Abbeel, O.; and Zaremba, W. 2017. Hindsight experience replay. Advances in neural information processing systems, 30

    2017

  3. [3]

    Bonnard, B.; Caillau, J.-B.; and Tr \'e lat, E. 2007. Second order optimality conditions in the smooth case and applications in optimal control. ESAIM: Control, Optimisation and Calculus of Variations, 13(2): 207--236

    work page 2007

  4. [4]

    Brandfonbrener, D.; Bietti, A.; Buckman, J.; Laroche, R.; and Bruna, J. 2022. When does return-conditioned supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, 35: 1542--1553

    work page 2022

  5. [5]

    M.; and Schneider, J

    Char, I.; Mehta, V.; Villaflor, A.; Dolan, J. M.; and Schneider, J. 2022. BATS: Best Action Trajectory Stitching. arXiv:2204.12026

    arXiv 2022

  6. [6]

    Cheikhi, D.; and Russo, D. 2023. On the statistical benefits of temporal difference learning. In International Conference on Machine Learning, 4269--4293. PMLR

    work page 2023

  7. [7]

    Chen, L.; Lu, K.; Rajeswaran, A.; Lee, K.; Grover, A.; Laskin, M.; Abbeel, P.; Srinivas, A.; and Mordatch, I. 2021. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084--15097

    2021

  8. [8]

    Chen, L.; Paleja, R.; and Gombolay, M. 2020. Learning from suboptimal demonstration via self-supervised reward regression. arXiv preprint arXiv:2010.11723

    arXiv 2020

Show all 59 references
  1. [9]

    Dargazany, A. 2021. DRL: Deep Reinforcement Learning for Intelligent Robot Control--Concept, Literature, and Future. arXiv preprint arXiv:2105.13806

    2021 arXiv

  2. [10]

    Ding, Y.; Florensa, C.; Abbeel, P.; and Phielipp, M. 2019. Goal-conditioned imitation learning. Advances in neural information processing systems, 32

    2019

  3. [11]

    Emmons, S.; Eysenbach, B.; Kostrikov, I.; and Levine, S. 2021. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751

    2021 arXiv

  4. [12]

    Eysenbach, B.; Salakhutdinov, R.; and Levine, S. 2020. C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909

    2020 arXiv

  5. [13]

    Eysenbach, B.; Zhang, T.; Levine, S.; and Salakhutdinov, R. R. 2022. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35: 35603--35620

    2022

  6. [14]

    Fujimoto, S.; Hoof, H.; and Meger, D. 2018. Addressing function approximation error in actor-critic methods. In International conference on machine learning, 1587--1596. PMLR

    2018

  7. [15]

    M.; Eysenbach, B.; and Levine, S

    Ghosh, D.; Gupta, A.; Reddy, A.; Fu, J.; Devin, C. M.; Eysenbach, B.; and Levine, S. 2021. Learning to Reach Goals via Iterated Supervised Learning. In International Conference on Learning Representations

    2021

  8. [16]

    Ghugare, R.; Geist, M.; Berseth, G.; and Eysenbach, B. 2024. Closing the Gap between TD Learning and Supervised Learning--A Generalisation Point of View. arXiv preprint arXiv:2401.11237

    2024 arXiv

  9. [17]

    Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; and Courville, A. C. 2017. Improved training of wasserstein gans. Advances in neural information processing systems, 30

    2017

  10. [18]

    Hansen, N.; and Wang, X. 2021. Generalization in Reinforcement Learning by Soft Data Augmentation. arXiv:2011.13389

    2021 arXiv

  11. [19]

    Hejna, J.; Gao, J.; and Sadigh, D. 2023. Distance Weighted Supervised Learning for Offline Interaction Data. arXiv preprint arXiv:2304.13774

    2023 arXiv

  12. [20]

    Kaelbling, L. P. 1993. Learning to achieve goals. In IJCAI, volume 2, 1094--8. Citeseer

    1993

  13. [21]

    Kahveci, N. E. 2007. Robust Adaptive Control For Unmanned Aerial Vehicles. Ph.D. thesis, University of Southern California

    2007

  14. [22]

    Kalashnikov, D.; Varley, J.; Chebotar, Y.; Swanson, B.; Jonschkowski, R.; Finn, C.; Levine, S.; and Hausman, K. 2021. MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale. arXiv:2104.08212

    2021 arXiv

  15. [23]

    Ke, L.; Zhang, Y.; Deshpande, A.; Srinivasa, S.; and Gupta, A. 2024. CCIL : Continuity-Based Data Augmentation for Corrective Imitation Learning. In The Twelfth International Conference on Learning Representations

    2024

  16. [24]

    Kostrikov, I.; Nair, A.; and Levine, S. 2021. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169

    2021 arXiv

  17. [25]

    Kostrikov, I.; Yarats, D.; and Fergus, R. 2021. Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. arXiv:2004.13649

    2021 arXiv

  18. [26]

    Levine, S.; Kumar, A.; Tucker, G.; and Fu, J. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643

    2020 arXiv

  19. [27]

    Li, W.; and Todorov, E. 2004. Iterative linear quadratic regulator design for nonlinear biological movement systems. In First International Conference on Informatics in Control, Automation and Robotics, volume 2, 222--229. SciTePress

    2004

  20. [28]

    P.; Hunt, J

    Lillicrap, T. P.; Hunt, J. J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; and Wierstra, D. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971

    2015 arXiv

  21. [29]

    Liu, M.; Zhu, M.; and Zhang, W. 2022. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.08299

    2022 arXiv

  22. [30]

    M.; Zhang, K.; and Schölkopf, B

    Lu, C.; Huang, B.; Wang, K.; Hernández-Lobato, J. M.; Zhang, K.; and Schölkopf, B. 2020. Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation. arXiv:2012.09092

    2020 arXiv

  23. [31]

    Lynch, C.; Khansari, M.; Xiao, T.; Kumar, V.; Tompson, J.; Levine, S.; and Sermanet, P. 2020. Learning latent plans from play. In Conference on robot learning, 1113--1132. PMLR

    2020

  24. [32]

    Y.; Yan, J.; Jayaraman, D.; and Bastani, O

    Ma, J. Y.; Yan, J.; Jayaraman, D.; and Bastani, O. 2022 a . Offline goal-conditioned reinforcement learning via f -advantage regression. Advances in Neural Information Processing Systems, 35: 310--323

    2022

  25. [33]

    J.; Yan, J.; Jayaraman, D.; and Bastani, O

    Ma, Y. J.; Yan, J.; Jayaraman, D.; and Bastani, O. 2022 b . How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f -Advantage Regression. arXiv preprint arXiv:2206.03023

    2022 arXiv

  26. [34]

    Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602

    2013 arXiv

  27. [35]

    V.; and Tak \'a c , M

    Oroojlooyjadid, A.; Nazari, M.; Snyder, L. V.; and Tak \'a c , M. 2022. A deep q-network for the beer game: Deep reinforcement learning for inventory optimization. Manufacturing & Service Operations Management, 24(1): 285--304

    2022

  28. [36]

    A.; and Ba, J

    Paster, K.; McIlraith, S. A.; and Ba, J. 2020. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:2012.02419

    2020 arXiv

  29. [37]

    A.; and Ba, J

    Paster, K.; Pitis, S.; McIlraith, S. A.; and Ba, J. 2023. Return Augmentation gives Supervised RL Temporal Compositionality

    2023

  30. [38]

    Plasencia-Salgueiro, A. d. J. 2023. Deep Reinforcement Learning for Autonomous Mobile Robot Navigation. In Artificial Intelligence for Robotics and Autonomous Systems Applications, 195--237. Springer

    2023

  31. [39]

    F.; Maximo, M

    Prudencio, R. F.; Maximo, M. R.; and Colombini, E. L. 2023. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems

    2023

  32. [40]

    Quiroga, F.; Hermosilla, G.; Farias, G.; Fabregas, E.; and Montenegro, G. 2022. Position control of a mobile robot through deep reinforcement learning. Applied Sciences, 12(14): 7194

    2022

  33. [41]

    Sarangapani, J. 2018. Neural network control of nonlinear discrete-time systems. CRC press

    2018

  34. [42]

    Schaul, T.; Horgan, D.; Gregor, K.; and Silver, D. 2015. Universal value function approximators. In International conference on machine learning, 1312--1320. PMLR

    2015

  35. [43]

    D.; et al

    Sestini, A.; Bagdanov, A. D.; et al. 2023. Deep reinforcement learning for the design and validation of modern computer games

    2023

  36. [44]

    M.; and Baillieul, J

    Seto, D.; Annaswamy, A. M.; and Baillieul, J. 1994. Adaptive control of nonlinear systems with a triangular structure. IEEE Transactions on Automatic Control, 39(7): 1411--1428

    1994

  37. [45]

    Sharifani, K.; and Amini, M. 2023. Machine Learning and Deep Learning: A Review of Methods and Applications. World Information Technology and Engineering Journal, 10(07): 3897--3904

    2023

  38. [46]

    Shorten, C.; and Khoshgoftaar, T. M. 2019. A survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 6: 1--48

    2019

  39. [47]

    Sikchi, H.; Chitnis, R.; Touati, A.; Geramifard, A.; Zhang, A.; and Niekum, S. 2024. Score Models for Offline Goal-Conditioned Reinforcement Learning. In The Twelfth International Conference on Learning Representations

    2024

  40. [48]

    Srinivas, A.; Laskin, M.; and Abbeel, P. 2020. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.04136

    2020 arXiv

  41. [49]

    K.; Shyam, P.; Mutz, F.; Ja \'s kowski, W.; and Schmidhuber, J

    Srivastava, R. K.; Shyam, P.; Mutz, F.; Ja \'s kowski, W.; and Schmidhuber, J. 2019. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02877

    2019 arXiv

  42. [50]

    Stone, A.; Ramirez, O.; Konolige, K.; and Jonschkowski, R. 2021. The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels. arXiv:2101.02722

    2021 arXiv

  43. [51]

    Wang, T.; Bao, X.; Clavera, I.; Hoang, J.; Wen, Y.; Langlois, E.; Zhang, S.; Zhang, G.; Abbeel, P.; and Ba, J. 2019. Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.02057

    2019 arXiv

  44. [52]

    Yamagata, T.; Khalil, A.; and Santos-Rodriguez, R. 2023. Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL

    2023

  45. [53]

    Yang, C.; Ma, X.; Huang, W.; Sun, F.; Liu, H.; Huang, J.; and Gan, C. 2019. Imitation learning from observations by minimizing inverse dynamics disagreement. arXiv preprint arXiv:1910.04417

    2019 arXiv

  46. [54]

    Yang, R.; Lu, Y.; Li, W.; Sun, H.; Fang, M.; Du, Y.; Li, X.; Han, L.; and Zhang, C. 2022. Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL. arXiv preprint arXiv:2202.04478

    2022 arXiv

  47. [55]

    a m \"a r \

    Yang, W.; Wang, H.; Cai, D.; Pajarinen, J.; and K \"a m \"a r \"a inen, J.-K. 2023. Swapped goal-conditioned offline reinforcement learning. arXiv preprint arXiv:2302.08865

    2023 arXiv

  48. [56]

    Yarats, D.; Fergus, R.; Lazaric, A.; and Pinto, L. 2021. Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning. arXiv:2107.09645

    2021 arXiv

  49. [57]

    D.; Maas, A

    Ziebart, B. D.; Maas, A. L.; Bagnell, J. A.; Dey, A. K.; et al. 2008. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, 1433--1438. Chicago, IL, USA

    2008

  50. [58]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.