NEUBAY uses Bayesian posteriors over world models with long-horizon planning to match or exceed conservative offline RL methods without explicit conservatism.
Combating the Compounding-Error Problem with a Multi-step Model
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
abstract
Model-based reinforcement learning is an appealing framework for creating agents that learn, plan, and act in sequential environments. Model-based algorithms typically involve learning a transition model that takes a state and an action and outputs the next state---a one-step model. This model can be composed with itself to enable predicting multiple steps into the future, but one-step prediction errors can get magnified, leading to unacceptable inaccuracy. This compounding-error problem plagues planning and undermines model-based reinforcement learning. In this paper, we address the compounding-error problem by introducing a multi-step model that directly outputs the outcome of executing a sequence of actions. Novel theoretical and empirical results indicate that the multi-step model is more conducive to efficient value-function estimation, and it yields better action selection compared to the one-step model. These results make a strong case for using multi-step models in the context of model-based reinforcement learning.
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
Advantage-guided diffusion (SAG and EAG) steers sampling in diffusion world models to higher-advantage trajectories, enabling policy improvement and better sample efficiency on MuJoCo tasks.
ADWM is an offline evaluation method that uses a policy-guided latent diffusion world model to rank LLM agent policies from pre-collected trajectories, reporting positive Spearman correlations on four benchmarks.
Return-conditional diffusion models for policies outperform offline RL on benchmarks by circumventing dynamic programming and enable constraint or skill composition.
HiL-ResRL trains a model-agnostic residual policy on VLA actions using human-guided online RL, achieving over 95% success rate after 1.5 hours of real-robot training.
Proposes a five-stage agentic AI framework for code review with human quality gates to maintain context, accountability, and team understanding.
citing papers explorer
-
Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
NEUBAY uses Bayesian posteriors over world models with long-horizon planning to match or exceed conservative offline RL methods without explicit conservatism.
-
Advantage-Guided Diffusion for Model-Based Reinforcement Learning
Advantage-guided diffusion (SAG and EAG) steers sampling in diffusion world models to higher-advantage trajectories, enabling policy improvement and better sample efficiency on MuJoCo tasks.
-
Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents
ADWM is an offline evaluation method that uses a policy-guided latent diffusion world model to rank LLM agent policies from pre-collected trajectories, reporting positive Spearman correlations on four benchmarks.
-
Is Conditional Generative Modeling all you need for Decision-Making?
Return-conditional diffusion models for policies outperform offline RL on benchmarks by circumventing dynamic programming and enable constraint or skill composition.
-
HiL-ResRL: A Model-Agnostic Finetuning Adapter via Human-in-the-loop Residual Reinforcement Learning
HiL-ResRL trains a model-agnostic residual policy on VLA actions using human-guided online RL, achieving over 95% success rate after 1.5 hours of real-robot training.
-
Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review
Proposes a five-stage agentic AI framework for code review with human quality gates to maintain context, accountability, and team understanding.
- Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination