RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences

Logan Mondal Bhamidipaty; Mykel Kochenderfer; Subramanian Ramamoorthy

REVIEW 3 major objections 5 minor 16 references

RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Binary preferences over imagined rollouts can directly train a world model's transition dynamics, and an uncertainty-directed querying scheme makes this practical for repairing model exploitation.

desk verdict Genuinely new idea — preferences supervising dynamics directly — with an honest but narrow proof-of-life; the RENEW advantage is real in some settings but not consistent enough for the headline claim. read the letter →

arxiv 2607.14180 v1 pith:GB2B3GSM submitted 2026-07-15 cs.LG cs.AI

Logan Mondal Bhamidipaty , Mykel Kochenderfer , Subramanian Ramamoorthy This is my paper · ORCID

classification cs.LGcs.AI

keywords worldmodelsmodelexploitationpreferencelearningBradley-Terryofflinereinforcementepistemicuncertaintydynamics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that human-style binary preferences over trajectory segments can supervise a world model's dynamics directly, not just its reward function. The authors propose Dynamics Learning from Human Feedback (DLHF), which plugs trajectory log-likelihood into the Bradley-Terry preference model, and RENEW, an active querying algorithm that uses epistemic uncertainty to focus preference labels on regions where the model is most likely to be exploited. They report that with a synthetic oracle, DLHF can learn simple world models from scratch, though at high label cost, and that RENEW's uncertainty targeting roughly halves final error compared to uniform querying, avoids catastrophic forgetting of pretrained dynamics, and reduces prediction error and epistemic uncertainty in pretrained world models using only 1,600 labels. If true, offline model-based RL could repair exploitable dynamics without expert demonstrations or environment interaction, offering an alternative to pessimistic conservative algorithms.

What carries the argument

The load-bearing identity is Eq. 2, which replaces the reward model with the dynamics model's own log-likelihood: P(σ0 ≻ σ1) = logistic(ℓθ(σ0) − ℓθ(σ1)). This makes the dynamics model itself the preference function, so binary labels supervise transitions directly. RENEW's other key mechanism is an epistemic-uncertainty estimator (ensemble disagreement) that is aggregated over trajectory segments and used to sample start states for preference querying, directing the finite label budget toward transitions where the model is most likely to be exploited; after each finetuning round, uncertainty is recomputed so subsequent queries adaptively target remaining weak spots.

What would settle it

Run the same Maze and Sokoban experiments with real human annotators instead of the synthetic oracle, asking them to compare single-step transitions from the model against ground-truth alternatives; if human label accuracy against ground truth drops significantly or if RENEW no longer reduces prediction error with 1,600 human labels, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Bradley-Terry preference framework, normally used to learn reward models, can instead learn transition dynamics by substituting the trajectory log-likelihood under the dynamics model for the reward model. Concretely, they define P(σ0 ≻ σ1) = logistic(ℓθ(σ0) − ℓθ(σ1)), where ℓθ is the sum of log-likelihoods of single transitions under the model. They show that optimizing this cross-entropy loss with preferences generated by a noiseless oracle can train latent-dynamics world models from scratch and finetune pretrained models to reduce exploitation. They further claim that actively sampling preference queries proportional to ensemble-disagreement epistemic

Load-bearing premise

The whole framework rests on the assumption that the synthetic oracle used in the experiments — labeling a transition pair by ℓ1 distance to the ground-truth next state — faithfully mimics what a real human would say when asked 'which transition is more physically plausible' at the single-transition granularity.

Editorial extensions

If this is right

Offline model-based RL can repair exploited world models using only binary preference labels, without collecting new demonstrations or querying the environment.
Active uncertainty-directed querying makes preference-based dynamics learning sample-efficient enough to be practical with a few thousand labels, whereas naive uniform querying requires orders of magnitude more.
Preference-based finetuning can reduce both prediction error and epistemic uncertainty in pretrained world models, directly targeting the conditions that lead to model exploitation.
The DLHF formulation naturally extends to latent dynamics models, so the approach applies to world models that operate over learned representations.
If the preference signal encodes physical plausibility rather than task success, it avoids the failure mode where reward preferences reinforce teleporting exploits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's experiments use a synthetic oracle that labels by ℓ1 distance to ground truth; whether the approach works with real human annotators, who may be noisy or reward-contaminated, remains an open empirical question that the authors themselves flag.
The single-transition (H=1) segment choice makes each preference query maximally simple, but in environments with smooth or high-dimensional observations, humans may find single-step differences hard to judge, suggesting longer segments may be necessary for real-world deployment.
A natural extension is to use DLHF-style preferences to finetune large generative video models, where human intuitive-physics judgments are already known to be reliable, rather than only small discrete grid worlds.
Combining DLHF with reward-preference RLHF could yield a single preference-elicitation pipeline that supervises both dynamics and rewards, potentially improving offline RL without any ground-truth environment access.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing to know: the paper has a real, clean idea. It replaces the reward model in Bradley-Terry preference learning with the world model's own trajectory log-likelihood, so pairwise human judgments of physical plausibility supervise the transition model directly. I have not seen that in prior work, and the related-work discussion supports the novelty claim. The formulation is simple, the experiments are transparent, and the code is linked. Credit where due: the proof-of-life in Section 3.1 shows binary preferences can learn simple discrete dynamics from scratch under a synthetic oracle, and Section 3.3 shows RENEW reduces prediction error and epistemic uncertainty on maze finetuning with 1,600 labels, with mostly clear confidence intervals. The authors also explicitly flag their main limitations, which is more than many papers do.

Now the soft spots, in proportion. The load-bearing assumption is the synthetic oracle: labels are assigned by ℓ1 distance to ground truth. The authors even admit DLHF under this oracle is "a diluted form of supervised learning." The practical motivation — humans relying on intuitive physics — is never tested. If real annotators are noisy, reward-contaminated, or unable to judge single transitions (H=1), the signal weakens or vanishes. So the paper demonstrates that a privileged oracle can supervise dynamics, not that human preferences can. That is a strictly weaker claim than the abstract suggests.

The second issue is that RENEW's sample-efficiency advantage is inconsistent. The ablation in Table 5 shows no consistent improvement over naive across K; on continuous control, naive beats RENEW at K=2. The advantage appears mainly in Section 3.3 maze finetuning and a couple of discrete cases. So "RENEW makes preferences practical" is not robustly supported. Also, "repairing exploitation" is measured through error and uncertainty proxies, not planner behavior or downstream task performance. Acceptable for a workshop paper, but it limits how strongly the results can be stated.

On the math and data: no circularity problem — preferences come from an external oracle, and the training loss does not equal the evaluation metric. Citations look honest. The paper is coherent on its own terms and the authors are not overselling in the conclusion, where they acknowledge the synthetic oracle and small environments.

Who gets value from this: anyone working on offline MBRL, preference-based learning, or world-model robustness. It deserves a serious referee — the idea is novel enough to warrant the time, even though the current evidence is narrow. I would send it to review with the expectation that the claims get scaled to what the experiments actually show.

Referee Report

3 major / 5 minor

Summary. The paper introduces DLHF (Dynamics Learning from Human Feedback), which replaces the reward model in the Bradley-Terry preference framework with the trajectory log-likelihood under a learned dynamics model (Eqs. 1--3), so that binary comparisons can in principle supervise transition dynamics directly. RENEW selects preference queries by epistemic uncertainty to target regions where the model is likely exploitable. Experiments on Jumanji grid environments and classic control tasks, using a synthetic oracle that labels pairs by ℓ1 distance to ground truth, show that naive DLHF can learn simple dynamics with very large label budgets, that RENEW gives mixed sample-efficiency gains over uniform querying, and that 1,600 preference labels can improve transition accuracy and reduce epistemic uncertainty of pretrained maze models. The conclusion acknowledges the synthetic-oracle limitation and small-scale environments.

Significance. If the preference signal works with real human annotators, this is a novel and potentially important direction for offline model-based RL: it would allow model repair without demonstrations, reward labels, or environment interaction. The formalization is clean, the code is released, and the ablations (candidate count K, tie handling, continuous control) are useful. However, the current evidence is a proof-of-life under a synthetic oracle; the human-preference premise and the robustness of the sample-efficiency advantage are not yet established. The paper is honest about its limitations, but the abstract and title claim more than the experiments support.

major comments (3)

[§3.1, §3.3, Appendix D] The entire empirical evaluation uses a synthetic oracle that labels preference pairs by ℓ1 distance to ground-truth transitions. This oracle has privileged access to the true transition, whereas the paper's motivating population is human annotators using intuitive physics. The conclusion itself admits 'our experiments use a synthetic oracle rather than real human annotators.' Without human elicitation or at least a noisy-oracle ablation, the central claim that preferences can supervise dynamics is only demonstrated for a privileged comparator. Since the DLHF loss (Eq. 3) is driven by these labels, this is a load-bearing, untested assumption. Please add a small human-subject study on a subset of environments, or a label-noise model (e.g., flip probability) and show that RENEW remains effective under realistic noise.
[§3.2, Table 5, Appendix G] The claim that 'RENEW improves sample efficiency over uniform querying' is not robust. In Table 2, the improvement is large for Sliding Tile 3×3 and Sokoban but negligible for Maze 5×5 and 10×10. Appendix F.1 (Table 5) shows near-parity and even worse RENEW results in several settings (e.g., sliding3 at K=2: 1.29 vs 1.26; maze10 at K=8: 0.041 vs 0.035). In continuous control (Table 7, K=2), naive DLHF outperforms RENEW on both MountainCar and Acrobot. The paper's own text in F.1 acknowledges that 'the gap between RENEW and the naive baseline is small at every value of K.' Please either restrict the sample-efficiency claim to discrete finetuning regimes with concentrated uncertainty or provide a systematic characterization of when active querying helps.
[§3.3, Conclusion] The paper frames RENEW as 'repairing model exploitation,' but §3.3 measures transition accuracy and epistemic uncertainty, not exploitability. No policy is trained or evaluated inside the repaired model, and no exploitability metric (e.g., planning return gap or the metric of Bhamidipaty et al. 2026) is reported. Reducing transition error in high-uncertainty cells is a reasonable proxy, but it does not establish that exploitation is repaired. Please add an end-to-end offline MBRL evaluation or explicitly reframe the contribution as reducing error and uncertainty in regions that give rise to exploitation.

minor comments (5)

[Eq. (2), Table 4] Equation (2) omits the Bradley-Terry temperature β, although Table 4 lists β = 1.0. Define the temperature in the equation or remove it from the hyperparameters.
[§3.3, Figure 4] The caption says 'per-cell transition error ... labeled by the number of wrong transitions'; the color scale is unclear. Please clarify whether the metric is count of incorrect next-state predictions per cell or a continuous error.
[Table 5, Footnote 5] Footnote 5 states that K=2 is optimal and 'the results reported here may understate RENEW's advantage,' but Table 5 shows the RENEW advantage is small and sometimes negative. The footnote overstates the implication of the ablation.
[Algorithm 1] Line 5 writes 'θ ← arg min θ LDLHF(θ;D≻)' which is not an update rule. State the optimizer and number of gradient steps per round.
[§3.1, Table 1] For 2048, both supervised and DLHF achieve ℓ1 error around 1.20, which is not close to zero. The surrounding text claims predictions match ground truth based on Figure 2; clarify that this holds only for the environments shown, not for all rows of Table 1.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: DLHF/RENEW's equations do not reduce to their inputs; the main caveats are an untested synthetic oracle and minor self-citations, which are validity concerns rather than circularity.

full rationale

The central derivation is not circular. Equation (1) defines trajectory log-likelihood under the learned dynamics model; Equation (2) plugs that log-likelihood into a Bradley-Terry logistic model; Equation (3) is the corresponding cross-entropy loss. The preference labels are external inputs (in the experiments, produced by a synthetic oracle), while the optimized quantity is the model's own log-likelihood. There is no step where Eq. 3 equals the label-generation process or where a fitted parameter is later reported as the predicted quantity. Evaluation metrics (validation ℓ1 error, transition accuracy, MSE) are computed against ground-truth dynamics independent of the training loss, so the reported improvements are not forced by construction. The active-querying loop uses epistemic uncertainty both to select queries and, in part, as a reported outcome (§3.3, Fig. 4). This is mildly self-referential because concentrating training on high-uncertainty transitions can be expected to reduce uncertainty there, but it is not a formal circularity: Table 3 reports transition accuracy, and uncertainty reduction is measured empirically rather than assumed. The paper itself flags the main weakness: 'Our experiments use a synthetic oracle rather than real human annotators' (Conclusion), and §3.1 admits 'under a synthetic oracle, DLHF is effectively a diluted form of supervised learning.' That is an external-validity limitation about whether human preferences would behave like the ℓ1-distance oracle, not a circularity in the derivation. Self-citations to Bhamidipaty et al. (2026) appear in framing statements about model exploitation and unexploitability, but the DLHF/RENEW formulation and experiments do not reduce to that work; no uniqueness theorem or ansatz is imported from the authors' prior papers. Overall, the claimed derivation is self-contained and externally evaluated, with only minor self-citational framing and a synthetic-oracle validity gap.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework rests on a small number of hand-chosen hyperparameters (H=1, K=2–4, E=3) and on the untested assumption that a synthetic ℓ1 oracle matches human judgments; the most consequential free 'parameter' is the preference budget itself, since DLHF needs ~1M labels to match 1K supervised transitions, and RENEW's advantage is sensitive to K and environment.

free parameters (5)

Segment horizon H = 1
Chosen for all experiments (Appendix D, Table 4) to make single-transition comparisons easy; if longer horizons are needed for distinguishable signals, reported label-efficiency numbers likely change.
Candidate count K = 2 in finetuning experiments, 4 in §3.3; ablation (Table 5) selects K=2 post hoc
K trades gradient steps against per-query signal; optimal value is environment/budget dependent, so 'RENEW is practical' depends on this hand-tuned choice.
Ensemble size E = 3
Ensemble disagreement is the epistemic-uncertainty estimator; E=3 is small and chosen by convention, not validated for uncertainty quality.
Batch size B = 64
Standard Adam batch; not fitted to data but a free choice affecting gradient steps per label.
Learning rates = 3e-4 (Jumanji), 3e-3 (classic control)
Hand-chosen per architecture; no sensitivity analysis reported.

assumptions (5)

domain assumption Pairwise preferences over transitions are determined by a scalar score that is a monotone function of true transition likelihood under the Bradley-Terry model.
Eq. 2/3 imports the RLHF preference model; if human realism judgments depend on non-monotone features (e.g., object identity, causality), this likelihood-as-score assumption fails.
ad hoc to paper Synthetic ℓ1 oracle labels are a faithful proxy for human physical-plausibility preferences.
Section 3.1 and 3.3 use ground-truth ℓ1 distance to generate labels; no human annotation experiments are reported, and the paper's conclusion lists this as a limitation.
domain assumption Ensemble disagreement identifies where the pretrained model is exploitable.
RENEW's query selection in §2.2 relies on this; supported by citations to MOPO/MOReL, but not verified for the specific environments/models here (continuous K=2 results suggest disagreement is noisy).
domain assumption Offline data is available only as a start-state pool, not as training signal.
Section 2.2 'RENEW finetunes exclusively on preference data'; if the training-data-free assumption is relaxed, simpler supervised methods dominate by 1000:1 labels.
ad hoc to paper H=1 comparisons are informative enough to supervise dynamics.
Appendix D sets H=1 for all main experiments; no horizon sensitivity shown, and the paper notes longer segments may be necessary in smoother environments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences." pith.science (2026). https://pith.science/paper/GB2B3GSM

@misc{pith2026260714180,
  author       = {Pith},
  title        = {Pith review of: RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GB2B3GSM}},
  note         = {Machine review of arXiv:2607.14180}
}

read the original abstract

World models are widely used in offline reinforcement learning (RL) to improve sample efficiency and generate experience beyond a fixed dataset. However, they are vulnerable to model exploitation where data coverage is thin. Prior work addresses this either by collecting more expert demonstrations, which is often expensive, unsafe, or unavailable, or by conservative algorithms that avoid uncertain regions, which limits generalization. We propose instead to repair exploitation directly using human preferences over imagined rollouts, leveraging the strong intuitive physics that allows humans to easily spot egregious dynamics hallucinations. We formalize this as Dynamics Learning from Human Feedback (DLHF), a Bradley-Terry preference loss over trajectory log-likelihoods under a learned dynamics model. Unfortunately, naive DLHF is sample inefficient, so we introduce RENEW, which uses epistemic uncertainty to focus finetuning where the model is most exploitable. We evaluate on several Jumanji and classic control environments and find that while naive DLHF requires an outsize preference budget, RENEW makes the framework practical by improving sample efficiency, limiting catastrophic forgetting, and reducing exploitation in pretrained world models. Taken together, our results provide initial evidence that preferences can supervise world model dynamics directly, offering a new approach to addressing exploitation in offline model-based RL.

Figures

Figures reproduced from arXiv: 2607.14180 by the authors.

**Figure 2.** Imagined rollouts from world models trained from preferences on three Jumanji environ [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Validation ℓ1 error versus cumulative preference labels for naive DLHF and RENEW across three Jumanji environments. Mean over 5 seeds; shaded regions indicate 95% CIs. to reach error comparable to 1K supervised transitions. We note that under a synthetic oracle, DLHF is effectively a diluted form of supervised learning [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: A single Maze 10×10 instance (matched 1600-label budget). Top: per-cell transition error (green cells are correct, warmer cells are mispredictions labeled by the number of wrong transitions). Bottom: per-cell epistemic uncertainty (darker = lower/better). Both finetune…

**Figure 5.** Figure 5: Naive DLHF versus RENEW on MountainCar-v0 and Acrobot-v1 at [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 3 linked inside Pith

[3]

Mean± 95% CI over 5 seeds. Env Method Exclude ties Include ties maze10 Naive.023±.002.021±.001 RENEW.022±.002.022±.001 sliding3 Naive1.26±.031.27±.02 RENEW1.32±.071.27±.07 sokoban Naive.115±.025.086±.018 RENEW.078±.020.085±.025 RENEW, excluding ties helps onsokobanbut including ties helps onsliding3. No consistent pattern emerges, and most differences fal...

2022
[6]

Offline learning from demonstrations and unlabeled experience.arXiv preprint arXiv:2011.13885,

Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Ay- tar, Misha Denil, Nando De Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience.arXiv preprint arXiv:2011.13885,

arXiv 2011
[8]

B Related Work Learning dynamics from preferences.Several recent papers have detailed methods for improv- ing world model realism with human preferences

(6) While RLHF has typically been applied to learn reward functions or policies (Christiano et al., 2017; Ouyang et al., 2022), in this work we apply the preference learning framework to thedynamics model ˆTθ, using human feedback to improve the fidelity of learned world models rather than to specify task objectives. B Related Work Learning dynamics from ...

2017
[9]

is closest to our setup: it operates offline, learns a transition model, generates rollouts in simulation, and elicits preferences, but uses those preferences exclu- sively to learn a reward model while leaving the dynamics fixed. RENEW offers a third response to model exploitation: rather than avoiding uncertain regions or collecting additional demonstra...

2025
[10]

demonstrates Finding the Frame Workshop at RLC 2026 through 67,000 human annotations that these violations are both pervasive and reliably judged by humans. These findings motivate preference-based supervision: the gap between what MLE learns from limited data and what humans trivially recognize as wrong is precisely the gap RENEW tar- gets. RLVR-World (W...

2026
[11]

also identifies MLE as a misaligned training objective but addresses it with verifiable task-specific rewards that require ground-truth access. RENEW instead uses binary preferences to close dynamics fidelity gaps without ground-truth states, targeting the complementary problem of physical plausibility rather than task alignment. C Architecture and Optimi...
[12]

Each block applies a depthwise3×3convolution, LayerNorm, a two-layer MLP with4×channel expansion and GELU activation, and a residual connection

as the residual unit throughout. Each block applies a depthwise3×3convolution, LayerNorm, a two-layer MLP with4×channel expansion and GELU activation, and a residual connection. The encoder maps discrete grid observations to a latent representation. Each cell is embedded via a learned tile embedding of dimension 32, producing a(rows,cols,32)grid. A linear...

2017
[13]

for continuous dynamics or Gumbel-Softmax relaxation (Jang et al., 2017; Maddison et al.,

2017

Show all 16 references

[16]

AtK= 2(1,562 gradient steps), naive outperforms RENEW on both environments. The state- action spaces are small (2 or 6 state dimensions, 3 actions), so uniform sampling provides adequate coverage, and ensemble disagreement is a noisier estimator of model error in continuous sp...

2026
[2004]

Whitammer, David Abel, Mykel J

Logan Mondal Bhamidipaty, Esmeralda S. Whitammer, David Abel, Mykel J. Kochenderfer, and Subramanian Ramamoorthy. Imperfect world models are exploitable.arXiv preprint arXiv:2605.15960,

arXiv
[2014]

Anatomy of a robotaxi crash: Lessons from the Cruise pedestrian dragging mishap

Finding the Frame Workshop at RLC 2026 Philip Koopman. Anatomy of a robotaxi crash: Lessons from the Cruise pedestrian dragging mishap. International Conference on Computer Safety, Reliability, and Security (SafeComp),

2026
[2019]

UCB/EECS-2019-98

Master’s thesis, EECS De- partment, University of California, Berkeley. UCB/EECS-2019-98. Renee Baillargeon. Infants’ reasoning about hidden objects: Evidence for event-general and event- specific expectations.Developmental Science, 7(4):391–414,

2019
[2020]

A Preliminaries We briefly review the necessary background to understand DLHF and RENEW

Finding the Frame Workshop at RLC 2026 Supplementary Materials The following content was not necessarily subject to peer review. A Preliminaries We briefly review the necessary background to understand DLHF and RENEW. A.1 Reinforcement Learning We consider a Markov decision pr...

2026
[2022]

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu

Software, version 0.0.4.http://github.com/RobertTLange/gymnax. Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643,

2005 arXiv
[2024]

•Maze10×10.A grid world where an agent navigates corridors to reach a goal position

environments used in our experimental evaluation. •Maze10×10.A grid world where an agent navigates corridors to reach a goal position. Walls are procedurally generated; the agent can move in four cardinal directions. •Sliding Tile5×5.A generalization of the classic 15-puzzle t...
[2025]

IntPhys 2019: A benchmark for visual intuitive physics under- standing.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5016–5025,

Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. IntPhys 2019: A benchmark for visual intuitive physics under- standing.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5016–5025,

2019

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [3]

Mean± 95% CI over 5 seeds. Env Method Exclude ties Include ties maze10 Naive.023±.002.021±.001 RENEW.022±.002.022±.001 sliding3 Naive1.26±.031.27±.02 RENEW1.32±.071.27±.07 sokoban Naive.115±.025.086±.018 RENEW.078±.020.085±.025 RENEW, excluding ties helps onsokobanbut including ties helps onsliding3. No consistent pattern emerges, and most differences fal...

2022

[2] [6]

Offline learning from demonstrations and unlabeled experience.arXiv preprint arXiv:2011.13885,

Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Ay- tar, Misha Denil, Nando De Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience.arXiv preprint arXiv:2011.13885,

arXiv 2011

[3] [8]

B Related Work Learning dynamics from preferences.Several recent papers have detailed methods for improv- ing world model realism with human preferences

(6) While RLHF has typically been applied to learn reward functions or policies (Christiano et al., 2017; Ouyang et al., 2022), in this work we apply the preference learning framework to thedynamics model ˆTθ, using human feedback to improve the fidelity of learned world models rather than to specify task objectives. B Related Work Learning dynamics from ...

2017

[4] [9]

is closest to our setup: it operates offline, learns a transition model, generates rollouts in simulation, and elicits preferences, but uses those preferences exclu- sively to learn a reward model while leaving the dynamics fixed. RENEW offers a third response to model exploitation: rather than avoiding uncertain regions or collecting additional demonstra...

2025

[5] [10]

demonstrates Finding the Frame Workshop at RLC 2026 through 67,000 human annotations that these violations are both pervasive and reliably judged by humans. These findings motivate preference-based supervision: the gap between what MLE learns from limited data and what humans trivially recognize as wrong is precisely the gap RENEW tar- gets. RLVR-World (W...

2026

[6] [11]

also identifies MLE as a misaligned training objective but addresses it with verifiable task-specific rewards that require ground-truth access. RENEW instead uses binary preferences to close dynamics fidelity gaps without ground-truth states, targeting the complementary problem of physical plausibility rather than task alignment. C Architecture and Optimi...

[7] [12]

Each block applies a depthwise3×3convolution, LayerNorm, a two-layer MLP with4×channel expansion and GELU activation, and a residual connection

as the residual unit throughout. Each block applies a depthwise3×3convolution, LayerNorm, a two-layer MLP with4×channel expansion and GELU activation, and a residual connection. The encoder maps discrete grid observations to a latent representation. Each cell is embedded via a learned tile embedding of dimension 32, producing a(rows,cols,32)grid. A linear...

2017

[8] [13]

for continuous dynamics or Gumbel-Softmax relaxation (Jang et al., 2017; Maddison et al.,

2017