Pith. sign in

REVIEW 4 major objections 5 minor 77 references

ProDVI: Programmatic Dynamics Priors for Value Network Initialization

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pretraining a critic's encoder on synthetic transitions generated by an LLM's coarse dynamics programs improves sample efficiency of model-free RL on Gym and DMControl, even when the programs are poor predictors.

desk verdict A useful and cheap warm-start for RL critics that consistently lifts aggregate returns, but the causal role of the LLM's domain knowledge is not tested against generic pretraining or the reset alone. read the letter →

arxiv 2608.06015 v1 pith:U5ZCRHIZ submitted 2026-08-06 cs.LG cs.AI

Xinwei Liu , Junyuan Liang , Jianting Zhang , Wuhui Chen This is my paper · ORCID
classification cs.LGcs.AI
keywords LLM-generateddynamicsprogramsvaluenetworkinitializationsampleefficiencyreinforcementlearningrepresentationpretrainingactor-criticsynthetictransitionspriors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a code-generating large language model can supply the knowledge needed to initialize a reinforcement-learning agent, replacing the usual prerequisites of offline trajectories, high-fidelity simulators, or meta-training tasks. The authors prompt an LLM to write coarse Python programs that approximate environment dynamics, use those programs to fabricate millions of synthetic transitions from random inputs, and pretrain the value network's state-action encoder to predict those pseudo-next-states. Online actor-critic training then starts from this representation. Across OpenAI Gym and DeepMind Control Suite locomotion tasks, the pretrained initialization improves aggregate returns and is especially strong at low environment-step budgets, even though the LLM programs are shown to be poor predictors of true next states. The paper takes this as evidence that approximate, even inaccurate, dynamics can supply a useful inductive bias for model-free RL.

What carries the argument

The load-bearing object is the LLM-generated dynamics-prior program $g_{\mathrm{LLM}}$, an executable Python function mapping an observation-action pair to an approximate next observation. ProDVI constructs it via three prompts: a Python-class description of the environment, a textual dynamics analysis, and a final code-generation prompt. From this program it synthesizes 7M transitions by sampling observations $\tilde{o} \sim \mathrm{clip}(\mathcal{N}(0,I), -10, 10)$ and actions $a \sim \mathcal{U}(a_{\min}, a_{\max})$, normalizing the outputs, and pretraining the encoder $f_\theta$ and decoder $d_\phi$ with the loss $\| d_\phi(f_\theta(\tilde{o}_t, a_t)) - \tilde{o}_{t+1} \|_2^2$. The pretrained weights $\bar\theta$ are then mixed with fresh initialization weights $\theta_0$ via $\theta_{\mathrm{init}} = \alpha \bar\theta + (1-\alpha)\theta_0$ with $\alpha = 0.5$, a shrink-and-perturb reset intended to preserve plasticity.

What would settle it

Ablate the dynamics content: use the same 7M Gaussian-clipped observation-action pairs but replace the LLM program's next-state outputs with random values or a constant, then run the full ProDVI pretraining and online RL. If the aggregate returns on Gym remain at the same level, the improvement does not come from the programmatic dynamics prior.

Watch

Extended reading notes

Core claim

ProDVI's central claim is that a dynamics prior elicited from an LLM's commonsense physics can be distilled into the state-action encoder of a value network before any real interaction, and that this initialization transfers even when the prior is quantitatively wrong. Concretely, AnonMethod+ProDVI outperforms AnonMethod with and without its auxiliary observation-prediction objective on aggregated normalized scores at 0.25M, 0.50M, and 1.00M environment steps on Gym, and on the dog and humanoid domains of DMControl; the same prior also improves a SAC-based variant. The paper reports that the generated programs are far from accurate simulators (e.g., Ant next-state MSE of 1.4e4), yet Table 4 shows that the pretrained encoder reaches lower training losses than a from-scratch encoder on a downstream dynamics-prediction task. The authors conclude that the benefit comes from informative dynamics structure in the learned hidden units, not from simulation fidelity.

Load-bearing premise

The whole benefit hinges on the transfer from synthetic inputs sampled off the real observation-action manifold: Gaussian-clipped observations and uniform actions, pushed through a program that is itself inaccurate, must teach features that generalize to real transitions.

Editorial extensions

If this is right

  • Model-free RL can be initialized without pre-collected datasets, high-fidelity simulators, or curated meta-learning task distributions; an LLM and environment metadata suffice.
  • Dynamics-prior programs are reusable across tasks governed by the same dynamics, since the prior is task-agnostic (e.g., one dog-domain program serves dog-run, dog-stand, dog-trot, and dog-walk).
  • The largest gains appear in the low-data regime, at 0.25M environment steps, meaning ProDVI primarily buys sample efficiency rather than asymptotic performance.
  • Inaccurate dynamics programs are sufficient: the improvement persists even when the program's next-state predictions are far from the true transitions.
  • The benefit transfers across actor-critic backbones, since both the TD3-based AnonMethod and a SAC-based variant improve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A plausible reading is that the synthetic inputs, being sampled broadly from a Gaussian rather than the real state-action manifold, act as a form of exploration or data augmentation that forces the encoder to learn smooth features; one could test this by keeping $g_{\mathrm{LLM}}$ but sampling inputs on-manifold and checking whether gains shrink.
  • The shrink-and-perturb interpolation at $\alpha = 0.5$ suggests that fully trusting the pretrained features hurts plasticity; an implicit prediction is that the optimal $\alpha$ depends on how long pretraining ran and on the distance between the synthetic and real manifolds.
  • If the mechanism is generic structure rather than physics, ProDVI might generalize to other pretraining functions such as random smooth programs or analytic functions, and the LLM's role could be replaced by any generator of structured input-output mappings.
  • Because the prior is injected only into the encoder, a natural extension would be to apply the same distillation to the policy network or to model-based world models, where the synthetic transitions could seed the learner's own predictive model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ProDVI, a method that prompts a code-generating LLM to produce executable Python functions encoding coarse hypotheses about environment dynamics, uses these functions to generate synthetic transitions from random Gaussian observations and uniform actions, pretrains the state-action encoder of an actor-critic value network with an auxiliary next-observation prediction loss, applies a shrink-and-perturb reset to the pretrained parameters, and then runs online RL. Experiments on five Gym locomotion tasks and seven DMControl dog/humanoid tasks compare AnonMethod and an SAC variant with and without ProDVI, reporting aggregate improvements in normalized score and IQM at 0.25M, 0.5M, and 1.0M environment steps, plus robustness across two additional LLM-generated dynamics programs. The authors acknowledge in Section 6 that the generated programs are not guaranteed to capture informative task-relevant dynamics regularities.

Significance. If the central claim is established, ProDVI is a novel and potentially useful resource-light initialization mechanism: it requires only an LLM and environment metadata, not pre-collected datasets, high-fidelity simulators, or meta-training task distributions. The paper has several strengths: 5-seed runs with 95% bootstrap confidence intervals, a two-stage sensitivity study, transfer experiments to a SAC-based backbone, per-task tables, prompt templates, and source code in the supplement. However, the causal claim that LLM-encoded commonsense and domain knowledge is the active ingredient is not yet supported, because the experimental design bundles synthetic pretraining, the shrink-and-perturb reset, and the semantic content of the generated programs. The paper's own Table 3 and Conclusion concede that the programs are inaccurate and unguaranteed, which makes the missing content-free control particularly load-bearing for the central claim.

major comments (4)
  1. [Section 5.4, Tables 3 and 4; Section 5.2] The experiments never isolate the semantic content of the LLM-generated program g_LLM from two other interventions bundled into ProDVI: pretraining on 7M synthetic transitions and the shrink-and-perturb reset of Eq. (5). The paper itself cites D'Oro et al. (2023) in Section 4.2, whose results show that resetting parameters can improve sample efficiency by itself. Section 5.5 changes only the LLM-generated program, and all three programs encode physically plausible structure, so it cannot rule out the hypothesis that any smooth synthetic pretraining target, or the reset alone, produces the same gains. Please add content-free controls consisting of a random program, a constant program, and no-pretraining with the reset only, and report RL returns at the same environment-step budgets.
  2. [Section 5.1, Appendix Tables 2-3] The evidence that the LLM prior transfers from off-manifold synthetic inputs to the real state-action manifold is indirect. Table 3 shows that the generated programs are far from accurate dynamics models (e.g., Ant MSE 1.4e4, MAPE 8.3e4%), and Table 4 reports only dynamics-prediction training losses after ProDVI initialization. Those lower losses are compatible with generic pretraining on any smooth function and do not establish that the LLM's domain knowledge is the active ingredient that produces the RL gains in Table 1. Please connect the dynamics-prediction losses to RL returns, ideally through the content-free control conditions, or provide a direct measure of representation transfer to the real manifold.
  3. [Tables 7 and 8, Section 5.2] The default hyperparameters N_syn=7M and alpha=0.5 are selected through a sensitivity study on Gym, and the headline Gym results in Table 1 are reported under this same choice. Although the paper states that the procedure is not an exhaustive grid search, selecting hyperparameters on the same benchmark used for the main claims is a selection loop. The authors should either include Gym results for at least one alternative sensible setting from the sensitivity study (e.g., 5M transitions or alpha=0.8) in the main table, or demonstrate that the qualitative conclusions are unchanged across the tested settings.
  4. [Section 5.2] The aggregate improvements conceal per-task regressions. For example, with AnonMethod at 0.25M environment steps, dog-run is 170 [161,181] without ProDVI versus 162 [154,170] with ProDVI, and dog-trot is 250 [225,276] versus 224 [198,251]. These differences are within overlapping confidence intervals, but the text in Section 5.2 should not imply uniform per-task improvement. Please report the number of tasks improved at each budget, discuss the regression cases, or provide a paired statistical test over seeds to support the aggregate claim.
minor comments (5)
  1. [Appendix B] The Deep-TD3-normalized score uses random scores, but the random scores in Appendix D are not clearly defined as average returns under random actions; please state the number of episodes and action distribution used for the random reference.
  2. [Appendix F] The phrase 'Observation Encoder Structure: identity function' in the AnonMethod hyperparameter table is ambiguous; clarify whether the normalized observation is concatenated directly with the action before the state-action encoder.
  3. [Appendix D] The prompt templates for GPT-5.5 are included, but the actual generated dynamics programs for all tasks are not; including the generated code for each task would substantially improve reproducibility, especially since GPT-5.5 is proprietary and may change over time.
  4. [Section 6] Typo: 'We use the-v5 version' should read 'We use the v5 version'.
  5. [Section 6] The Conclusion states that program quality is constrained by LLM capabilities, but the paper does not quantify how performance degrades when the LLM produces a poor program; noting this limitation is good, but a concrete failure analysis would strengthen the paper.

Circularity Check

1 steps flagged · score 3.0 of 10

No equational circularity; central gains are empirical, but the Gym default configuration is selected on Gym, creating a mild selection loop, and the backbone is an anonymous self-citation.

  1. fitted input called prediction [Section 5.1 (Experimental Setup) and Appendix E, Tables 2-3]
    "ProDVI uses 7M synthetic transitions and sets α=0.5 as the default configuration for all experiments. This configuration is selected through a simple two-stage sensitivity study on Gym. We first fix α=1.0 and vary the number of synthetic transitions among 1,3,5,7,10M, where 7M achieves the best overall performance. We then fix the number of synthetic transitions to 7M and vary α among {1.0,0.8,0.5,0.2}, finding that α=0.5 provides strong performance."

    The default configuration is chosen by maximizing aggregated Gym scores, and the main Gym evidence (Table 1) is then produced under that same configuration. Consequently, the reported Gym improvements are post-selection rather than independent predictions: the hyperparameters were fitted to the same benchmark used to support the claim. This is a mild selection loop, not an equational derivation—other settings also improve and DMControl is held out—so it weakens but does not force the central claim.

full rationale

ProDVI's derivation chain is not circular by construction: the pretraining objective (Eq. 2) predicts synthetic g_LLM outputs, while the reported success metric is real-environment return; the paper itself shows g_LLM is inaccurate (Table 3), so the result is an empirical transfer claim rather than an identity. The main circularity-adjacent issue is hyperparameter selection: 7M and α=0.5 are chosen via a Gym sensitivity study, and the same Gym results are then offered as evidence of improvement; this is a selection loop, though not a forced one. A second concern is that the backbone AnonMethod and NormMethod are anonymous self-citations, but ProDVI's internal comparisons are relative to that same backbone, so the self-citation is not load-bearing for the mechanism. The most significant experimental gap—absence of a content-free control (random or constant program) to isolate LLM domain knowledge—is a missing-control issue, not circularity. Overall, the central claims are empirical and independently testable; score 3 reflects the mild Gym selection loop rather than any equation-level reduction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The two core ProDVI hyperparameters (7M transitions and alpha=0.5) are tuned on the same Gym benchmark where the main results are reported. No invented physical entities are introduced. The method's transfer signal rests on the assumption that random synthetic inputs evaluated by an inaccurate LLM program still yield useful representations for real dynamics.

free parameters (2)
  • Number of synthetic transitions N_syn = 7,000,000
    Selected via a two-stage sensitivity study on Gym with alpha fixed to 1.0 and N_syn in {1M, 3M, 5M, 7M, 10M}; 7M achieved the best overall Gym performance and was then used for all main experiments, including the same Gym benchmark.
  • Interpolation coefficient alpha = 0.5
    Selected after fixing N_syn=7M on Gym, varying alpha in {1.0, 0.8, 0.5, 0.2}; alpha=0.5 gave the best overall performance and was then used for all experiments, including the same Gym benchmark used for selection.
assumptions (4)
  • domain assumption The observation o_t can be treated as the Markovian state s_t.
    Section 3 states this and restricts the method to state-based settings; if observations are non-Markovian, the dynamics-prior pretraining target is ill-posed.
  • ad hoc to paper Random Gaussian observations clipped to [-10,10] and uniform actions produce inputs on which an LLM-written dynamics program yields outputs whose structure is informative for the real control task.
    Section 4.2 constructs the synthetic dataset this way via Equations (3) and (4); the paper validates the transfer only indirectly through training losses in Table 4, not by a distributional argument.
  • domain assumption AnonMethod and NormMethod (Anonymous 2026) are valid, strong model-free RL components and match the appendix descriptions.
    The entire evaluation uses AnonMethod as the backbone; the citation is suppressed for double-blind review and no public implementation is linked, so the baseline cannot be independently checked.
  • ad hoc to paper LLM-generated dynamics programs with large prediction errors still contain task-relevant structure.
    Section 5.4 Table 3 shows high MSE and MAPE on real transitions, but the authors assume residual structure helps; this is tested via training losses in Table 4 rather than via RL returns.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProDVI: Programmatic Dynamics Priors for Value Network Initialization." pith.science (2026). https://pith.science/paper/U5ZCRHIZ

@misc{pith2026260806015,
  author       = {Pith},
  title        = {Pith review of: ProDVI: Programmatic Dynamics Priors for Value Network Initialization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U5ZCRHIZ}},
  note         = {Machine review of arXiv:2608.06015}
}
read the original abstract

Deep Reinforcement Learning (RL) is notoriously sample inefficient. One contributing factor is that RL agents are typically initialized from scratch, forcing them to acquire task-relevant knowledge through online interaction. Existing approaches obtain informative initializations through pre-collected datasets, high-fidelity simulators, or meta-learning over related tasks, but these prerequisites may be difficult to access or even unavailable. In this paper, we propose Programmatic Dynamics Priors for Value Network Initialization (ProDVI), a framework that leverages the commonsense and domain knowledge encoded in large language models to initialize RL agents without relying on these resources. Specifically, ProDVI prompts a code-generating language model to produce executable Python functions that encode coarse hypotheses about environment dynamics. These functions are then used to generate synthetic transitions. Based on these transitions, we construct an auxiliary dynamics prediction objective to pretrain the state-action encoder of the value network in an actor-critic framework. The learned representation provides dynamics-aware inductive biases before online RL begins. Importantly, the generated programs are used only for representation pretraining and are not required to faithfully simulate the target environment. While the generated programs may be inaccurate, their induced initialization can be corrected through online learning from real transitions and rewards. Experiments on OpenAI Gym and DeepMind Control Suite tasks show that ProDVI can effectively improve the sample efficiency of model-free RL algorithms.

Figures

Figures reproduced from arXiv: 2608.06015 by the authors.

Figure 1
Figure 1. Overview of ProDVI. (1) Environment information is formatted as a Python class-style prompt for the LLM. The [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 1
Figure 1. Per-task and aggregate learning curves on the Gym benchmark for AnonMethod w/o Aux, AnonMethod, and their [PITH_FULL_IMAGE:figures/full_fig_p020_1.png] view at source ↗
Figure 2
Figure 2. Per-task and aggregate learning curves on the DMControl tasks for AnonMethod w/o Aux, AnonMethod, and their [PITH_FULL_IMAGE:figures/full_fig_p021_2.png] view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Per-task and aggregate learning curves on the Gym benchmark for AnonMethod-SAC w/o Aux, AnonMethod-SAC, [PITH_FULL_IMAGE:figures/full_fig_p022_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 45 canonical work pages

  1. [1]

    The Tenth International Conference on Learning Representations,

    Denis Yarats and Rob Fergus and Alessandro Lazaric and Lerrel Pinto , title =. The Tenth International Conference on Learning Representations,. 2022 , url =

    work page 2022

  2. [2]

    9th International Conference on Learning Representations,

    Denis Yarats and Ilya Kostrikov and Rob Fergus , title =. 9th International Conference on Learning Representations,. 2021 , url =

    2021

  3. [3]

    The Thirteenth International Conference on Learning Representations,

    Scott Fujimoto and Pierluca D'Oro and Amy Zhang and Yuandong Tian and Michael Rabbat , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    work page 2025

  4. [4]

    Devon Hjelm and Aaron C

    Max Schwarzer and Ankesh Anand and Rishab Goel and R. Devon Hjelm and Aaron C. Courville and Philip Bachman , title =. 9th International Conference on Learning Representations,. 2021 , url =

    2021

  5. [5]

    Max Schwarzer and Johan S. Obando. Bigger, Better, Faster: Human-level Atari with human-level efficiency , booktitle =. 2023 , url =

    work page 2023

  6. [6]

    Self-Predictive Dynamics for Generalization of Vision-based Reinforcement Learning

    Kyungsoo Kim and Jeongsoo Ha and Yusung Kim , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.05418 , eprinttype =. 2506.05418 , timestamp =

    work page Pith review arXiv doi:10.48550/arxiv.2506.05418 2025

  7. [7]

    Bootstrap Your Own Latent -

    Jean. Bootstrap Your Own Latent -. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , year =

    2020

  8. [8]

    Bellemare , editor =

    Carles Gelada and Saurabh Kumar and Jacob Buckman and Ofir Nachum and Marc G. Bellemare , editor =. DeepMDP: Learning Continuous Latent Space Models for Representation Learning , booktitle =. 2019 , url =

    work page 2019

Show all 77 references
  1. [9]

    Nature , pages=

    Mastering diverse control tasks through world models , author=. Nature , pages=. 2025 , publisher=

    2025

  2. [10]

    The Twelfth International Conference on Learning Representations,

    Nicklas Hansen and Hao Su and Xiaolong Wang , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  3. [11]

    1998 , publisher=

    Reinforcement learning: An introduction , author=. 1998 , publisher=

    1998

  4. [12]

    Lillicrap and Martin A

    Yuval Tassa and Yotam Doron and Alistair Muldal and Tom Erez and Yazhe Li and Diego de Las Casas and David Budden and Abbas Abdolmaleki and Josh Merel and Andrew Lefrancq and Timothy P. Lillicrap and Martin A. Riedmiller , title =. CoRR , volume =. 2018 , url =. 1801.00690 , t...

    2018 arXiv

  5. [13]

    Learning Representations via a Robust Behavioral Metric for Deep Reinforcement Learning , booktitle =

    Jianda Chen and Sinno Jialin Pan , editor =. Learning Representations via a Robust Behavioral Metric for Deep Reinforcement Learning , booktitle =. 2022 , url =

    2022

  6. [14]

    MICo: Improved representations via sampling-based state similarity for Markov decision processes , booktitle =

    Pablo Samuel Castro and Tyler Kastner and Prakash Panangaden and Mark Rowland , editor =. MICo: Improved representations via sampling-based state similarity for Markov decision processes , booktitle =. 2021 , url =

    2021

  7. [15]

    The Twelfth International Conference on Learning Representations,

    Guozheng Ma and Lu Li and Sen Zhang and Zixuan Liu and Zhen Wang and Yixin Chen and Li Shen and Xueqian Wang and Dacheng Tao , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  8. [16]

    Why do We Need Large Batchsizes in Contrastive Learning?

    Changyou Chen and Jianyi Zhang and Yi Xu and Liqun Chen and Jiali Duan and Yiran Chen and Son Tran and Belinda Zeng and Trishul Chilimbi , editor =. Why do We Need Large Batchsizes in Contrastive Learning?. Advances in Neural Information Processing Systems 35: Annual Conferenc...

    2022

  9. [17]

    Contrastive Learning as Goal-Conditioned Reinforcement Learning , booktitle =

    Benjamin Eysenbach and Tianjun Zhang and Sergey Levine and Ruslan Salakhutdinov , editor =. Contrastive Learning as Goal-Conditioned Reinforcement Learning , booktitle =. 2022 , url =

    2022

  10. [18]

    The Thirteenth International Conference on Learning Representations,

    Grace Liu and Michael Tang and Benjamin Eysenbach , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

    2025

  11. [19]

    Policy-Independent Behavioral Metric-Based Representation for Deep Reinforcement Learning , booktitle =

    Weijian Liao and Zongzhang Zhang and Yang Yu , editor =. Policy-Independent Behavioral Metric-Based Representation for Deep Reinforcement Learning , booktitle =. 2023 , url =. doi:10.1609/AAAI.V37I7.26052 , timestamp =

    2023 doi

  12. [20]

    Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learning , journal =

    Leiji Zhang and Zeyu Wang and Xin Li and Yao. Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learning , journal =. 2025 , url =. doi:10.48550/ARXIV.2507.18519 , eprinttype =. 2507.18519 , timestamp =

    2025 doi

  13. [21]

    Scott Fujimoto and Wei. For. Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , year =

    2023

  14. [22]

    Jha and Toshisada Mariyama and Daniel Nikovski , title =

    Kei Ota and Tomoaki Oiki and Devesh K. Jha and Toshisada Mariyama and Daniel Nikovski , title =. Proceedings of the 37th International Conference on Machine Learning,. 2020 , url =

    2020

  15. [23]

    Lillicrap and Jimmy Ba and Mohammad Norouzi , title =

    Danijar Hafner and Timothy P. Lillicrap and Jimmy Ba and Mohammad Norouzi , title =. 8th International Conference on Learning Representations,. 2020 , url =

    2020

  16. [24]

    Lillicrap and Mohammad Norouzi and Jimmy Ba , title =

    Danijar Hafner and Timothy P. Lillicrap and Mohammad Norouzi and Jimmy Ba , title =. 9th International Conference on Learning Representations,. 2021 , url =

    2021

  17. [25]

    Addressing Function Approximation Error in Actor-Critic Methods , booktitle =

    Scott Fujimoto and Herke van Hoof and David Meger , editor =. Addressing Function Approximation Error in Actor-Critic Methods , booktitle =. 2018 , url =

    2018

  18. [26]

    Emanuel Todorov and Tom Erez and Yuval Tassa , title =. 2012. 2012 , url =. doi:10.1109/IROS.2012.6386109 , timestamp =

    2012

  19. [27]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library , booktitle =

    Adam Paszke and Sam Gross and Francisco Massa and Adam Lerer and James Bradbury and Gregory Chanan and Trevor Killeen and Zeming Lin and Natalia Gimelshein and Luca Antiga and Alban Desmaison and Andreas K. PyTorch: An Imperative Style, High-Performance Deep Learning Library ,...

    2019

  20. [28]

    Lukasz Kaiser and Mohammad Babaeizadeh and Piotr Milos and Blazej Osinski and Roy H. Campbell and Konrad Czechowski and Dumitru Erhan and Chelsea Finn and Piotr Kozakowski and Sergey Levine and Afroz Mohiuddin and Ryan Sepassi and George Tucker and Henryk Michalewski , title =...

    2020

  21. [29]

    Bellemare and Yavar Naddaf and Joel Veness and Michael Bowling , title =

    Marc G. Bellemare and Yavar Naddaf and Joel Veness and Michael Bowling , title =. J. Artif. Intell. Res. , volume =. 2013 , url =. doi:10.1613/JAIR.3912 , timestamp =

    2013 doi

  22. [30]

    iQRL - Implicitly Quantized Representations for Sample-efficient Reinforcement Learning , journal =

    Aidan Scannell and Kalle Kujanp. iQRL - Implicitly Quantized Representations for Sample-efficient Reinforcement Learning , journal =. 2024 , url =. doi:10.48550/ARXIV.2406.02696 , eprinttype =. 2406.02696 , timestamp =

    2024 doi arXiv

  23. [31]

    Ruijie Zheng and Xiyao Wang and Yanchao Sun and Shuang Ma and Jieyu Zhao and Huazhe Xu and Hal Daum. Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 202...

    2023

  24. [32]

    Riedmiller , title =

    David Silver and Guy Lever and Nicolas Heess and Thomas Degris and Daan Wierstra and Martin A. Riedmiller , title =. Proceedings of the 31th International Conference on Machine Learning,. 2014 , url =

    2014

  25. [33]

    Lillicrap and David Silver , title =

    Julian Schrittwieser and Ioannis Antonoglou and Thomas Hubert and Karen Simonyan and Laurent Sifre and Simon Schmitt and Arthur Guez and Edward Lockhart and Demis Hassabis and Thore Graepel and Timothy P. Lillicrap and David Silver , title =. Nat. , volume =. 2020 , url =. doi...

    2020 doi

  26. [34]

    Dueling Network Architectures for Deep Reinforcement Learning , booktitle =

    Ziyu Wang and Tom Schaul and Matteo Hessel and Hado van Hasselt and Marc Lanctot and Nando de Freitas , editor =. Dueling Network Architectures for Deep Reinforcement Learning , booktitle =. 2016 , url =

    2016

  27. [35]

    Temporal Difference Learning for Model Predictive Control , booktitle =

    Nicklas Hansen and Hao Su and Xiaolong Wang , editor =. Temporal Difference Learning for Model Predictive Control , booktitle =. 2022 , url =

    2022

  28. [36]

    Riedmiller , title =

    Volodymyr Mnih and Koray Kavukcuoglu and David Silver and Alex Graves and Ioannis Antonoglou and Daan Wierstra and Martin A. Riedmiller , title =. CoRR , volume =. 2013 , url =. 1312.5602 , timestamp =

    2013 arXiv

  29. [37]

    Ayoub Echchahed and Pablo Samuel Castro , title =. Trans. Mach. Learn. Res. , volume =. 2025 , url =

    2025

  30. [38]

    An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay , booktitle =

    Scott Fujimoto and David Meger and Doina Precup , editor =. An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay , booktitle =. 2020 , url =

    2020

  31. [39]

    7th International Conference on Learning Representations,

    Ilya Loshchilov and Frank Hutter , title =. 7th International Conference on Learning Representations,. 2019 , url =

    2019

  32. [40]

    Rainbow: Combining Improvements in Deep Reinforcement Learning , booktitle =

    Matteo Hessel and Joseph Modayil and Hado van Hasselt and Tom Schaul and Georg Ostrovski and Will Dabney and Dan Horgan and Bilal Piot and Mohammad Gheshlaghi Azar and David Silver , editor =. Rainbow: Combining Improvements in Deep Reinforcement Learning , booktitle =. 2018 ,...

    2018 doi

  33. [41]

    Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search , booktitle =

    Nicola Dainese and Matteo Merler and Minttu Alakuijala and Pekka Marttinen , editor =. Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search , booktitle =. 2024 , url =

    2024

  34. [42]

    WorldCoder, a Model-Based

    Hao Tang and Darren Key and Kevin Ellis , editor =. WorldCoder, a Model-Based. Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024 , year =

    2024

  35. [43]

    Cal-QL: Calibrated Offline

    Mitsuhiko Nakamoto and Simon Zhai and Anikait Singh and Max Sobol Mark and Yi Ma and Chelsea Finn and Aviral Kumar and Sergey Levine , editor =. Cal-QL: Calibrated Offline. Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing...

    2023

  36. [44]

    Martin and Mariano Phielipp and Chelsea Finn , editor =

    Rafael Rafailov and Kyle Beltran Hatch and Victor Kolev and John D. Martin and Mariano Phielipp and Chelsea Finn , editor =. Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA,. 2023 , url =

    2023

  37. [45]

    Thirty-Eighth

    Jiaheng Feng and Mingxiao Feng and Haolin Song and Wengang Zhou and Houqiang Li , editor =. Thirty-Eighth. 2024 , url =. doi:10.1609/AAAI.V38I11.29083 , timestamp =

    2024 doi

  38. [46]

    Xue Bin Peng and Marcin Andrychowicz and Wojciech Zaremba and Pieter Abbeel , title =. 2018. 2018 , url =. doi:10.1109/ICRA.2018.8460528 , timestamp =

    2018

  39. [47]

    Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021 , year =

    Ashish Kumar and Zipeng Fu and Deepak Pathak and Jitendra Malik , editor =. Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021 , year =. doi:10.15607/RSS.2021.XVII.011 , timestamp =

    2021 doi

  40. [48]

    Ilija Radosavovic and Tete Xiao and Bike Zhang and Trevor Darrell and Jitendra Malik and Koushil Sreenath , title =. Sci. Robotics , volume =. 2024 , url =. doi:10.1126/SCIROBOTICS.ADI9579 , timestamp =

    2024 doi

  41. [49]

    Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , booktitle =

    Chelsea Finn and Pieter Abbeel and Sergey Levine , editor =. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , booktitle =. 2017 , url =

    2017

  42. [50]

    7th International Conference on Learning Representations,

    Jonas Rothfuss and Dennis Lee and Ignasi Clavera and Tamim Asfour and Pieter Abbeel , title =. 7th International Conference on Learning Representations,. 2019 , url =

    2019

  43. [51]

    The Twelfth International Conference on Learning Representations,

    Lirui Wang and Yiyang Ling and Zhecheng Yuan and Mohit Shridhar and Chen Bao and Yuzhe Qin and Bailin Wang and Huazhe Xu and Xiaolong Wang , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  44. [52]

    RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation , booktitle =

    Yufei Wang and Zhou Xian and Feng Chen and Tsun. RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation , booktitle =. 2024 , url =

    2024

  45. [53]

    2024 , url =

    Pushkal Katara and Zhou Xian and Katerina Fragkiadaki , title =. 2024 , url =. doi:10.1109/ICRA57147.2024.10610566 , timestamp =

    2024

  46. [54]

    Rusu and Joel Veness and Marc G

    Volodymyr Mnih and Koray Kavukcuoglu and David Silver and Andrei A. Rusu and Joel Veness and Marc G. Bellemare and Alex Graves and Martin A. Riedmiller and Andreas Fidjeland and Georg Ostrovski and Stig Petersen and Charles Beattie and Amir Sadik and Ioannis Antonoglou and Hel...

    2015 doi

  47. [55]

    David Silver and Aja Huang and Chris J. Maddison and Arthur Guez and Laurent Sifre and George van den Driessche and Julian Schrittwieser and Ioannis Antonoglou and Vedavyas Panneershelvam and Marc Lanctot and Sander Dieleman and Dominik Grewe and John Nham and Nal Kalchbrenner...

    2016 doi

  48. [56]

    Challenges of Real-World Reinforcement Learning , journal =

    Gabriel Dulac. Challenges of Real-World Reinforcement Learning , journal =. 2019 , url =. 1904.12901 , timestamp =

    2019 arXiv

  49. [57]

    EfficientZero

    Shengjie Wang and Shaohuai Liu and Weirui Ye and Jiacheng You and Yang Gao , editor =. EfficientZero. Forty-first International Conference on Machine Learning,. 2024 , url =

    2024

  50. [58]

    9th International Conference on Learning Representations,

    Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt , title =. 9th International Conference on Learning Representations,. 2021 , url =

    2021

  51. [59]

    CoRR , volume =

    OpenAI , title =. CoRR , volume =. 2023 , url =. doi:10.48550/ARXIV.2303.08774 , eprinttype =. 2303.08774 , timestamp =

    2023 doi arXiv

  52. [60]

    CoRR , volume =

    OpenAI , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2601.03267 , eprinttype =. 2601.03267 , timestamp =

    2026 doi arXiv

  53. [61]

    Eureka: Human-Level Reward Design via Coding Large Language Models , booktitle =

    Yecheng Jason Ma and William Liang and Guanzhi Wang and De. Eureka: Human-Level Reward Design via Coding Large Language Models , booktitle =. 2024 , url =

    2024

  54. [62]

    CoRR , volume =

    Thang Duong and Minglai Yang and Chicheng Zhang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.10861 , eprinttype =. 2505.10861 , timestamp =

    2025 doi arXiv

  55. [63]

    Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning , booktitle =

    Yun Qu and Yuhang Jiang and Boyuan Wang and Yixiu Mao and Cheems Wang and Chang Liu and Xiangyang Ji , editor =. Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning , booktitle =. 2025 , url =. doi:10.1609/AAAI.V39I19.34213 , timestamp =

    2025 doi

  56. [64]

    LLM-Explorer:

    Qianyue Hao and Yiwen Song and Qingmin Liao and Jian Yuan and Yong Li , editor =. LLM-Explorer:. Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico...

    2025

  57. [65]

    CoRR , volume =

    Xinwei Liu and Junyuan Liang and Zicong Hong and Jianting Zhang and Wuhui Chen , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2606.21297 , eprinttype =. 2606.21297 , timestamp =

    2026 doi arXiv

  58. [66]

    Balis and Gianluca De Cola and Tristan Deleu and Manuel Goul

    Mark Towers and Ariel Kwiatkowski and John U. Balis and Gianluca De Cola and Tristan Deleu and Manuel Goul. Gymnasium:. Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, Dece...

    2025

  59. [67]

    Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , booktitle =

    Tuomas Haarnoja and Aurick Zhou and Pieter Abbeel and Sergey Levine , editor =. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , booktitle =. 2018 , url =

    2018

  60. [68]

    The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks , booktitle =

    Nicholas Carlini and Chang Liu and. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks , booktitle =. 2019 , url =

    2019

  61. [69]

    Extracting Training Data from Large Language Models , booktitle =

    Nicholas Carlini and Florian Tram. Extracting Training Data from Large Language Models , booktitle =. 2021 , url =

    2021

  62. [70]

    Feder Cooper and Daphne Ippolito and Christopher A

    Milad Nasr and Javier Rando and Nicholas Carlini and Jonathan Hayase and Matthew Jagielski and A. Feder Cooper and Daphne Ippolito and Christopher A. Choquette. Scalable Extraction of Training Data from Aligned, Production Language Models , booktitle =. 2025 , url =

    2025

  63. [71]

    LLM-Empowered State Representation for Reinforcement Learning , booktitle =

    Boyuan Wang and Yun Qu and Yuhang Jiang and Jianzhun Shao and Chang Liu and Wenming Yang and Xiangyang Ji , editor =. LLM-Empowered State Representation for Reinforcement Learning , booktitle =. 2024 , url =

    2024

  64. [72]

    Machado , editor =

    Zaheer Abbas and Rosie Zhao and Joseph Modayil and Adam White and Marlos C. Machado , editor =. Loss of Plasticity in Continual Deep Reinforcement Learning , booktitle =. 2023 , url =

    2023

  65. [73]

    Fernando Hernandez

    Shibhansh Dohare and J. Fernando Hernandez. Loss of plasticity in deep continual learning , journal =. 2024 , url =. doi:10.1038/S41586-024-07711-7 , timestamp =

    2024 doi

  66. [74]

    Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier , booktitle =

    Pierluca D'Oro and Max Schwarzer and Evgenii Nikishin and Pierre. Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier , booktitle =. 2023 , url =

    2023

  67. [75]

    Spectral Normalisation for Deep Reinforcement Learning: An Optimisation Perspective , booktitle =

    Florin Gogianu and Tudor Berariu and Mihaela Rosca and Claudia Clopath and Lucian Busoniu and Razvan Pascanu , editor =. Spectral Normalisation for Deep Reinforcement Learning: An Optimisation Perspective , booktitle =. 2021 , url =

    2021

  68. [76]

    Gomes and Kilian Q

    Johan Bjorck and Carla P. Gomes and Kilian Q. Weinberger , editor =. Towards Deeper Deep Reinforcement Learning with Spectral Normalization , booktitle =. 2021 , url =

    2021

  69. [77]

    2026 , note =

    Anonymous , title =. 2026 , note =

    2026

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.