Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Are Expressive Models Truly Necessary for Offline RL?

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that a 2-layer MLP recursive planner can match or beat heavy transformers and diffusion planners on long-horizon offline RL, in minutes of training.

desk verdict RSP is a genuinely new, simple algorithm with strong long-horizon D4RL results, but the paper's explanation of why it works is not fully validated and a few reproducibility issues need fixing before the numbers are fully trustworthy. read the letter →

arxiv 2412.11253 v1 pith:N2YOVR52 submitted 2024-12-15 cs.LG cs.AI

Guan Wang , Haoyi Niu , Jianxiong Li , Li Jiang , Jianming Hu , Xianyuan Zhan This is my paper · ORCID
classification cs.LGcs.AI
keywords offlinereinforcementlearninggoal-conditionedsupervisedrecursiveskip-stepplanningsubgoalpredictioncompoundingerrormodelefficiencyD4RLlong-horizon
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large expressive models are necessary for offline reinforcement learning with long horizons, and answers no. It argues that a lightweight recursive planner can keep sequential modeling error low enough that shallow two-layer MLPs outperform transformer- and diffusion-based sequence models. The proposed Recursive Skip-Step Planning (RSP) predicts coarse future subgoals at exponentially shrinking horizons and conditions a goal-conditioned policy on them, entirely in a supervised-learning framework. The claim is that this combination yields state-of-the-art D4RL scores on multi-stage long-horizon tasks while training in roughly 180 seconds and acting in under a millisecond.

What carries the argument

The central object is the recursive skip-step dynamics stack. For recursion depth $N$ and top-level skip $K$, the dataset is relabeled so the $n$-th model predicts the subgoal at horizon $K(n)=K/2^n$, conditioned on all higher-level subgoals $\kappa^{(n-1)}$; the lowest-level subgoal then conditions the policy $\pi(a_t | \kappa^{(N)})$. At evaluation, Equation 6 runs the stack top-down, each model consuming the previous model's output. This reduces long-horizon planning to an exponentially shrinking chain of coarse-to-fine predictions, which is why a few recursive calls can replace thousands of single-step rollouts and why rollout RMSE falls as the skip step grows from 4 to 64.

What would settle it

On AntMaze-Ultra, run RSP once with the lowest-level subgoal taken from the ground-truth relabeled dataset and once with the recursively predicted subgoal, holding everything else fixed. If success rate rises sharply with the ground-truth subgoal, the recursive predictors are not closing the loop and the long-horizon advantage depends on policy conditioning rather than on accurate planning.

Watch

Extended reading notes

Core claim

The central discovery is that coarse-grained recursive planning removes the need for high-capacity models in long-horizon offline RL. RSP relabels each transition with subgoal ground truths at horizons $K/2^n$, learns a stack of $N$ skip-step dynamics models and one goal-conditioned policy by maximum likelihood, and at test time predicts subgoals recursively from the highest to the lowest level before generating an action. Measured on D4RL, this scheme reaches new state-of-the-art scores on AntMaze and Kitchen, including about 80 normalized return on AntMaze-Ultra tasks where transformer and diffusion baselines mostly fail, while using only two-layer 1024-unit MLPs. The authors' interpretation is that long horizons are best handled by planning few, coarse, recursively refined subgoals instead of many fine-grained steps, so compounding error stops growing with trajectory length.

Load-bearing premise

The method assumes that subgoal predictors trained on real future states stay accurate when they are fed their own predictions at test time; if the predicted subgoals drift away from the training distribution, the recursive chain and the policy that consumes it will degrade.

Editorial extensions

If this is right

  • RSP reaches state-of-the-art normalized scores on D4RL AntMaze across all nine datasets, including about 80 on the extremely long AntMaze-Ultra tasks where most baselines score below 40 and several score zero.
  • Because RSP uses only two-layer MLPs, its training time is about 180 seconds and inference under 1 ms at recursion depth one, orders of magnitude faster than transformer- or diffusion-based sequence planners of comparable score.
  • Coarse-grained skip-step prediction lowers rollout RMSE as the skip step grows from 4 to 64, meaning cumulative error falls as the planning step becomes coarser rather than rising with sequence length.
  • Adding recursion depth beyond two gives little benefit on most tasks, because each extra prediction level adds its own error, so depth two is the recommended cost-performance point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's RMSE analysis suggests a testable scaling property: as the top-level skip $K$ grows, the number of recursive calls needed to cover a fixed horizon shrinks logarithmically, so the scheme may extend to horizons far beyond 3000 steps; the paper does not run that test.
  • A direct robustness probe left untested is to inject small noise into the lowest-level predicted subgoal and measure the success drop, which would quantify how much of RSP's edge depends on subgoal accuracy versus the policy's tolerance to inaccurate goals.
  • Training the recursive dynamics models on their own predictions, a closed-loop objective, would target the weakest assumption directly and could make the method robust to drift; this is an extension, not a claim of the paper.
  • The method is evaluated in near-deterministic simulated environments; how single-point subgoal predictions behave under stochastic transitions is an open question, since a fixed future state may not be well-defined when dynamics are random.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Recursive Skip-Step Planning (RSP), a goal-conditioned supervised learning method for offline RL that uses shallow 2-layer MLPs for recursive coarse-grained sub-goal prediction and a goal-conditioned policy. The method relabels transitions with skip-step sub-goal ground truths, trains N dynamics models to predict sub-goals at increasing horizons, and at test time feeds predicted sub-goals recursively into the policy. The authors report state-of-the-art results on D4RL AntMaze and Kitchen, with training time around 180 seconds and inference under 1 ms, and claim that large expressive models are not necessary for long-horizon offline RL.

Significance. If the empirical claims hold, RSP is a significant result: it would show that a simple recursive coarse-grained planning scheme with MLPs can outperform Transformer- and diffusion-based sequence models on long-horizon tasks at a fraction of the compute. The paper has notable strengths: it reports 10-seed means with standard deviations, includes ablations on recursion depth and skip-step horizon, measures training/inference time in isolation, and compares against a wide set of baselines. The main weakness is that the recursive operating regime is not evaluated in closed loop, and several reproducibility details are missing or inconsistent; these issues currently prevent full confidence in the central claim.

major comments (3)
  1. [Recursive Skip-Step Planning (Eqs. 5-6); Discussions] The central technical claim requires that dynamics models trained on ground-truth relabeled sub-goals (Eq. 5) remain accurate when their outputs are fed back through Eq. 6 at test time. The paper never trains on predicted sub-goals, and the Discussions section explicitly states that the RMSE analysis in Fig. 3 'assumes that an oracle policy model perfectly follow the skip-step state predictions.' Consequently, the claimed 'accurate dynamics consistency' is validated only for ground-truth-conditioned rollouts, not for the actual recursive operating regime. This is a load-bearing gap; please add closed-loop rollout evaluation (e.g., feeding predicted sub-goals to the policy and measuring task success) and/or a DAgger-style training correction that conditions on predicted sub-goals.
  2. [Training Cost and Inference Latency; Discussions; Additional Experimental Details] The paper is internally inconsistent about the main hyperparameters. Discussions and Additional Experimental Details state k=32 and N=1 are used across all tasks, but the Training Cost and Inference Latency section reports 'RSP with horizon k = 8 and recursion depth N = 1, 4' and Fig. 1 and its caption show RSP (N=1) and RSP (N=4). Please reconcile these values and report the exact configuration for each table row, since the claimed 180-second training time depends on a specific N and k.
  3. [Baselines; Table 1] Table 1 reports scores for HIQL in AntMaze and Kitchen, but HIQL is never introduced in the Baselines paragraph, no citation is given for the reproduced numbers, and it is not clear whether those numbers are from the original paper or self-reproduced. Since HIQL is a strong baseline and appears in the tasks where RSP claims large gains, the source and protocol must be specified.
minor comments (5)
  1. [Abstract and Conclusion] The claim of 'new SOTA performances on the D4RL benchmark' is too broad: Table 1 shows RSP is not SOTA on MuJoCo average (76.4 vs TT's 78.9) and ties TAP on Adroit average (67.0). Please restrict the SOTA claim to long-horizon AntMaze and Kitchen tasks.
  2. [Abstract (code availability)] The code availability statement 'Code is available at RSP JAX' is not a usable link; please provide a repository URL or remove the claim.
  3. [Recursive Skip-Step Planning, Eq. (4)-(5)] Equation (4) defines κ^(n) but Eq. (5) conditions on κ^(n-1) to predict s_{K^(n-1)}; the indices are inconsistent with the textual description of the recursion and should be clarified.
  4. [Ablation Studies, Fig. 4(a)] Figure 4(a) is titled with four task types but has no legend distinguishing the curves; please add a legend or label the curves directly.
  5. [Table 1 footnote] The footnote mentions that IQL scores on AntMaze-v2 are taken from DOGE, but no corresponding citation of DOGE appears in the reference list; please add the reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RSP's predictions are supervised fits to ground-truth relabeled sub-goals, benchmark scores come from external D4RL evaluation, and the few self-citations are not load-bearing.

full rationale

The paper's derivation chain is self-contained rather than circular. Its dynamics models are trained by maximum likelihood on ground-truth relabeled sub-goals (Eq. 5) and its policy is trained by maximum likelihood on ground-truth sub-goal conditioning (Eq. 7); the evaluation-time recursion (Eq. 6) feeds predicted sub-goals into the policy, which is a standard autoregressive generation procedure, not a definitional identification. No equation in the paper defines the predicted sub-goal as the fitted target by construction, and no benchmark score is obtained by renaming a fitted parameter. The RMSE analysis in Fig. 3 is explicitly acknowledged in the Discussions section to assume an oracle policy and therefore does not overclaim; this is a stated limitation about train/eval distribution shift, not a circular step. The paper's central results are measured against the external D4RL benchmark and against independently published baselines, so the SOTA claims rest on empirical comparison rather than on the paper's own assumptions. Self-citations to prior work by the same group (e.g., Zhan et al. 2022; Cheng et al. 2024; Niu et al. 2022) appear only as motivational or contextual references about model-based offline RL and are not invoked to justify the correctness of RSP or to forbid alternative designs. Hyperparameters N and K are chosen from ablations on AntMaze tasks, but the reported scores are not constructed from those choices by definition; the ablations themselves are presented as empirical sensitivity analyses. Overall, the derivation chain is independent of its conclusions, so no circularity is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two user-chosen hyperparameters (N and K), standard GCSL relabeling assumptions, and two unproved domain assumptions: that recursive coarse-graining reduces error and that predicted subgoals stay in-distribution at test time. No new physical or model entities are introduced; sub-goals are future states from the dataset.

free parameters (2)
  • Recursion depth N = 1 (ablated over 1-4)
    Chosen after ablation on AntMaze-Ultra and fixed across all tasks. It controls the number of dynamics models and the final performance.
  • Skip-step horizon K (lowest-level k) = 32 (ablated over 1,4,8,16,32,64; some text/figures use k=8)
    Selected to balance long-horizon planning ability against policy conditioning difficulty. The choice is benchmark-driven and affects the main results.
assumptions (4)
  • domain assumption The offline dataset has enough trajectory coverage that hindsight relabeling makes any future state a meaningful goal for supervised learning.
    Invoked in the GCSL reformulation section; standard for this line of work but can fail on suboptimal or non-Markovian data.
  • domain assumption Recursive sub-goal prediction with coarser horizons reduces approximation error at each level.
    Stated in the Recursive Skip-Step Planning section and supported only by RMSE experiments (Fig. 3), not by a proof. Underlies the method's claimed mechanism.
  • domain assumption A 2-layer MLP can represent the environment's skip-step dynamics and goal-conditioned policy well enough on D4RL tasks.
    The central argument that expressive models are unnecessary depends on this capacity assumption. It is validated empirically but not theoretically.
  • domain assumption At evaluation, recursively predicted subgoals remain close enough to the training distribution of ground-truth subgoals.
    Dynamics models are trained only on ground-truth relabeled subgoals, but evaluation feeds them their own predictions (Eq. 6). No distribution-shift correction or data augmentation with predicted subgoals is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Expressive Models Truly Necessary for Offline RL?." pith.science (2026). https://pith.science/paper/N2YOVR52

@misc{pith2026241211253,
  author       = {Pith},
  title        = {Pith review of: Are Expressive Models Truly Necessary for Offline RL?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N2YOVR52}},
  note         = {Machine review of arXiv:2412.11253}
}
read the original abstract

Among various branches of offline reinforcement learning (RL) methods, goal-conditioned supervised learning (GCSL) has gained increasing popularity as it formulates the offline RL problem as a sequential modeling task, therefore bypassing the notoriously difficult credit assignment challenge of value learning in conventional RL paradigm. Sequential modeling, however, requires capturing accurate dynamics across long horizons in trajectory data to ensure reasonable policy performance. To meet this requirement, leveraging large, expressive models has become a popular choice in recent literature, which, however, comes at the cost of significantly increased computation and inference latency. Contradictory yet promising, we reveal that lightweight models as simple as shallow 2-layer MLPs, can also enjoy accurate dynamics consistency and significantly reduced sequential modeling errors against large expressive models by adopting a simple recursive planning scheme: recursively planning coarse-grained future sub-goals based on current and target information, and then executes the action with a goal-conditioned policy learned from data rela-beled with these sub-goal ground truths. We term our method Recursive Skip-Step Planning (RSP). Simple yet effective, RSP enjoys great efficiency improvements thanks to its lightweight structure, and substantially outperforms existing methods, reaching new SOTA performances on the D4RL benchmark, especially in multi-stage long-horizon tasks.

Figures

Figures reproduced from arXiv: 2412.11253 by the authors.

Figure 1
Figure 1. Existing GCSL methods trade off score performances for slow training and inference. However, RSP with recursion depth N = 1 significantly outperforms all baselines on Antmaze datasets while completing training in just 180 seconds and enjoying an inference latency of 0.33ms. N = 4 further enhances perfor￾mance with minimal cost, as the inference time increases to 0.48 ms and the training time to 271s, remaining order… view at source ↗
Figure 2
Figure 2. The illustration of Recursive Skip-Step Planning [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Root mean squared error (RMSE) between rollout sub-goal predictions and ground truths on [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablations on choices of recursion depth and horizon. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Skill Expansion and Composition in Parameter Space

    cs.LG 2025-02 conditional novelty 6.0 of 10

    PSEC shows that weighting and summing LoRA skill modules inside a diffusion policy network outperforms composing the same skills in action or noise space across D4RL, DSRL, DMC, and Meta-World tasks.

Reference graph

Works this paper leans on

57 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [32]

    Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10012--10022

    2021

  2. [1]

    Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning

    Ada, S. E.; Oztop, E.; and Ugur, E. 2023. Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning. arXiv preprint arXiv:2307.04726

    work page Pith review arXiv 2023

  3. [2]

    Ajay, A.; Du, Y.; Gupta, A.; Tenenbaum, J.; Jaakkola, T.; and Agrawal, P. 2022. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657

    arXiv 2022

  4. [3]

    Amos, B.; Stanton, S.; Yarats, D.; and Wilson, A. G. 2021. On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamics and Control, 6--20. PMLR

    work page 2021

  5. [4]

    An, G.; Moon, S.; Kim, J.-H.; and Song, H. O. 2021. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in Neural Information Processing Systems

    work page 2021

  6. [5]

    Bai, C.; Wang, L.; Yang, Z.; Deng, Z.-H.; Garg, A.; Liu, P.; and Wang, Z. 2021. Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning. In International Conference on Learning Representations

    work page 2021

  7. [6]

    Chen, H.; Lu, C.; Ying, C.; Su, H.; and Zhu, J. 2023. Offline reinforcement learning via high-fidelity generative behavior modeling. International Conference on Learning Representations

    work page 2023

  8. [7]

    Chen, L.; Lu, K.; Rajeswaran, A.; Lee, K.; Grover, A.; Laskin, M.; Abbeel, P.; Srinivas, A.; and Mordatch, I. 2021. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084--15097

    2021

Show all 57 references
  1. [8]

    Cheng, P.; Zhan, X.; Zhang, W.; Lin, Y.; Wang, H.; Jiang, L.; et al. 2024. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. Advances in Neural Information Processing Systems, 36

    2024

  2. [9]

    Emmons, S.; Eysenbach, B.; Kostrikov, I.; and Levine, S. 2021. RvS: What is Essential for Offline RL via Supervised Learning? arXiv preprint arXiv:2112.10751

    2021 arXiv

  3. [10]

    Emmons, S.; Eysenbach, B.; Kostrikov, I.; and Levine, S. 2022. RvS: What is Essential for Offline RL via Supervised Learning? International Conference on Learning Representations

    2022

  4. [11]

    Feng, X.; Jiang, L.; Yu, X.; Xu, H.; Sun, X.; Wang, J.; Zhan, X.; and Chan, W. K. V. 2022. Curriculum Goal-Conditioned Imitation for Offline Reinforcement Learning. IEEE Transactions on Games

    2022

  5. [12]

    Fu, J.; Kumar, A.; Nachum, O.; Tucker, G.; and Levine, S. 2020. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv preprint

    2020

  6. [13]

    Fujimoto, S.; and Gu, S. S. 2021. A Minimalist Approach to Offline Reinforcement Learning. ArXiv preprint

    2021

  7. [14]

    Fujimoto, S.; Meger, D.; and Precup, D. 2019. Off-Policy Deep Reinforcement Learning without Exploration. In International Conference on Machine Learning, 2052--2062

    2019

  8. [15]

    Garg, D.; Hejna, J.; Geist, M.; and Ermon, S. 2023. Extreme Q-Learning: MaxEnt RL without Entropy. arXiv preprint arXiv:2301.02328

    2023 arXiv

  9. [16]

    M.; Eysenbach, B.; and Levine, S

    Ghosh, D.; Gupta, A.; Reddy, A.; Fu, J.; Devin, C. M.; Eysenbach, B.; and Levine, S. 2021. Learning to Reach Goals via Iterated Supervised Learning. In International Conference on Learning Representations

    2021

  10. [17]

    G.; and Levine, S

    Hansen-Estruch, P.; Kostrikov, I.; Janner, M.; Kuba, J. G.; and Levine, S. 2023. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573

    2023 arXiv

  11. [18]

    Hu, J.; Sun, Y.; Huang, S.; Guo, S.; Chen, H.; Shen, L.; Sun, L.; Chang, Y.; and Tao, D. 2023 a . Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learning. arXiv preprint arXiv:2306.04875

    2023 arXiv

  12. [19]

    Hu, S.; Shen, L.; Zhang, Y.; and Tao, D. 2023 b . Graph Decision Transformer. arXiv preprint arXiv:2303.03747

    2023 arXiv

  13. [20]

    Janner, M.; Du, Y.; Tenenbaum, J.; and Levine, S. 2022. Planning with Diffusion for Flexible Behavior Synthesis. In International Conference on Machine Learning, 9902--9915. PMLR

    2022

  14. [21]

    Janner, M.; Fu, J.; Zhang, M.; and Levine, S. 2019. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32

    2019

  15. [22]

    Janner, M.; Li, Q.; and Levine, S. 2021. Offline Reinforcement Learning as One Big Sequence Modeling Problem. Advances in Neural Information Processing Systems

    2021

  16. [23]

    Jiang, Z.; Zhang, T.; Janner, M.; Li, Y.; Rockt \"a schel, T.; Grefenstette, E.; and Tian, Y. 2022. Efficient planning in a compact latent action space. arXiv preprint arXiv:2208.10291

    2022 arXiv

  17. [24]

    G.; Seraj, E.; and Gombolay, M

    Konan, S. G.; Seraj, E.; and Gombolay, M. 2023. Contrastive decision transformers. In Conference on Robot Learning, 2159--2169. PMLR

    2023

  18. [25]

    Kostrikov, I.; Fergus, R.; Tompson, J.; and Nachum, O. 2021. Offline Reinforcement Learning with Fisher Divergence Critic Regularization. In International Conference on Machine Learning, 5774--5783

    2021

  19. [26]

    Kostrikov, I.; Nair, A.; and Levine, S. 2021. Offline reinforcement learning with implicit q-learning. ArXiv preprint

    2021

  20. [27]

    Kumar, A.; Fu, J.; Soh, M.; Tucker, G.; and Levine, S. 2019. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. In Advances in Neural Information Processing Systems, 11761--11771

    2019

  21. [28]

    B.; and Levine, S

    Kumar, A.; Peng, X. B.; and Levine, S. 2019. Reward-conditioned policies. arXiv preprint arXiv:1912.13465

    2019 arXiv

  22. [29]

    Kumar, A.; Zhou, A.; Tucker, G.; and Levine, S. 2020. Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems

    2020

  23. [30]

    Levine, S.; Kumar, A.; Tucker, G.; and Fu, J. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv preprint

    2020

  24. [31]

    Li, J.; Zhan, X.; Xu, H.; Zhu, X.; Liu, J.; and Zhang, Y.-Q. 2023. When data geometry meets deep function: Generalizing offline reinforcement learning. In International Conference on Learning Representations

    2023

  25. [33]

    Lu, C.; Chen, H.; Chen, J.; Su, H.; Li, C.; and Zhu, J. 2023. Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning. arXiv preprint arXiv:2304.12824

    2023 arXiv

  26. [34]

    Lynch, C.; Khansari, M.; Xiao, T.; Kumar, V.; Tompson, J.; Levine, S.; and Sermanet, P. 2020. Learning latent plans from play. In Conference on robot learning

    2020

  27. [35]

    Lyu, J.; Ma, X.; Li, X.; and Lu, Z. 2022. Mildly conservative Q-learning for offline reinforcement learning. arXiv preprint arXiv:2206.04745

    2022 arXiv

  28. [36]

    A.; Hjelm, D.; Toshev, A.; and Susskind, J

    Mazoure, B.; Talbott, W.; Bautista, M. A.; Hjelm, D.; Toshev, A.; and Susskind, J. 2023. Value function estimation using conditional diffusion models for control. arXiv preprint arXiv:2306.07290

    2023 arXiv

  29. [37]

    Niu, H.; Chen, Q.; Liu, T.; Li, J.; Zhou, G.; ZHANG, Y.; HU, J.; and Zhan, X. 2024. x TED : Cross-Domain Adaptation via Diffusion-Based Trajectory Editing. In NeurIPS 2024 Workshop on Open-World Agents

    2024

  30. [38]

    Niu, H.; Sharma, S.; Qiu, Y.; Li, M.; Zhou, G.; Hu, J.; and Zhan, X. 2022. When to trust your simulator: dynamics-aware hybrid offline-and-online reinforcement learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 36599--36612

    2022

  31. [39]

    Schmidhuber, J. 2019. Reinforcement Learning Upside Down: Don't Predict Rewards--Just Map Them to Actions. arXiv preprint arXiv:1912.02875

    2019 arXiv

  32. [40]

    Van Den Oord, A.; Vinyals, O.; et al. 2017. Neural discrete representation learning. Advances in neural information processing systems, 30

    2017

  33. [41]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  34. [42]

    R.; Huang, Z.; Pande, S.; Dolan, J

    Villaflor, A. R.; Huang, Z.; Pande, S.; Dolan, J. M.; and Schneider, J. 2022. Addressing optimism bias in sequence modeling for reinforcement learning. In international conference on machine learning, 22270--22283. PMLR

    2022

  35. [43]

    Wang, H.; Lin, Y.; Han, S.; and Lv, K. 2023. Offline Reinforcement Learning with Diffusion-Based Behavior Cloning Term. In International Conference on Knowledge Science, Engineering and Management, 267--278. Springer

    2023

  36. [44]

    Wang, K.; Zhao, H.; Luo, X.; Ren, K.; Zhang, W.; and Li, D. 2022. Bootstrapped transformer for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 34748--34761

    2022

  37. [45]

    Wang, X.; Xu, H.; Zheng, Y.; and Zhan, X. 2024. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. Advances in Neural Information Processing Systems, 36

    2024

  38. [46]

    J.; and Zhou, M

    Wang, Z.; Hunt, J. J.; and Zhou, M. 2022. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193

    2022 arXiv

  39. [47]

    Wu, Y.; Tucker, G.; and Nachum, O. 2019. Behavior Regularized Offline Reinforcement Learning. arXiv preprint arXiv:1911.11361

    2019 arXiv

  40. [48]

    Wu, Y.-H.; Wang, X.; and Hamaya, M. 2023. Elastic Decision Transformer. arXiv preprint arXiv:2307.02484

    2023 arXiv

  41. [49]

    Xiao, C.; Wang, H.; Pan, Y.; White, A.; and White, M. 2023. The In-Sample Softmax for Offline Reinforcement Learning. arXiv preprint arXiv:2302.14372

    2023 arXiv

  42. [50]

    Xu, H.; Jiang, L.; Li, J.; Yang, Z.; Wang, Z.; Chan, V. W. K.; and Zhan, X. 2023. Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization. arXiv preprint arXiv:2303.15810

    2023 arXiv

  43. [51]

    Xu, H.; Jiang, L.; Li, J.; and Zhan, X. 2022. A Policy-Guided Imitation Approach for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems

    2022

  44. [52]

    Zhan, X.; Xu, H.; Zhang, Y.; Zhu, X.; Yin, H.; and Zheng, Y. 2022. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 4680--4688

    2022

  45. [53]

    Zhan, X.; Zhu, X.; and Xu, H. 2022. Model-based offline planning with trajectory pruning. In International Joint Conference on Artificial Intelligence (IJCAI)

    2022

  46. [54]

    Zhang, T.; Janner, M.; Li, Y.; Rockt \"a schel, T.; Grefenstette, E.; Tian, Y.; et al. 2022. Efficient Planning in a Compact Latent Action Space. In International Conference on Learning Representations

    2022

  47. [55]

    E.; Zhan, X.; and Liu, J

    Zheng, Y.; Li, J.; Yu, D.; Yang, Y.; Li, S. E.; Zhan, X.; and Liu, J. 2024. Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model. In International Conference on Learning Representations

    2024

  48. [56]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.