Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models

Aleksandr I. Panov; Aleksei Staroverov; Dmitriy Poyarkov

REVIEW 3 major objections 5 minor 22 references

Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding offline supervision to PPO lets vision-language-action models match a twice-longer reinforcement learning run in half the environment steps, preserving out-of-distribution gains.

desk verdict The paper's core sample-efficiency result is credible for environment steps; the 'half the training budget' claim needs to be scoped to that axis, or backed with compute numbers. read the letter →

arxiv 2607.19399 v1 pith:IX3GR6IV submitted 2026-07-06 cs.LG cs.AIcs.CV

Dmitriy Poyarkov , Aleksei Staroverov , Aleksandr I. Panov This is my paper · ORCID

classification cs.LGcs.AIcs.CV

keywords reinforcementlearningvision-language-actionmodelsofflinesupervisionPPOKLregularizationbehaviorcloningout-of-distributiongeneralizationsampleefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether offline supervision can be folded into online RL fine-tuning of large vision-language-action models so that training is faster without losing the out-of-distribution robustness that pure RL provides. The authors test two simple modifications of Proximal Policy Optimization (PPO): one that penalizes divergence from a frozen supervised reference policy (RefKL), and one that adds behavior cloning on the offline demonstration data (DataBC). On an out-of-distribution benchmark for a large VLA model, both variants outperform standard PPO at the same environment-step budget, and the reference-guided variant matches the final performance of a twice-as-long PPO run. The paper concludes that offline supervision acts as an optimization prior that accelerates the early phase of RL rather than as a replacement for it. The practical payoff is that hybrid training can make RL fine-tuning of large VLA policies more viable while keeping RL's generalization edge.

What carries the argument

The central object is a modified PPO objective with an auxiliary offline-supervision term. RefKL adds a Kullback-Leibler divergence penalty between the current policy and a frozen reference policy (the supervised model) on sampled minibatches; DataBC adds a behavior-cloning log-likelihood loss on the offline demonstration batch. Both use a curriculum schedule that holds the auxiliary coefficient constant for the first phase, linearly anneals it, and then removes it, allowing the policy to finish training with pure RL. This mechanism supplies a supervised prior during the phase when the on-policy RL signal is weak, and the paper argues it preserves RL's out-of-distribution gains while improvi

What would settle it

Run the 1M-step RefKL and the 2M-step standard PPO on identical hardware and measure total wall-clock time and FLOPs. If the guided run is not faster than the longer PPO run, then the efficiency claim, which the paper bases on environment steps, is falsified in practical terms.

Watch

Extended reading notes

Core claim

The central claim is that when offline supervision is included in the RL objective—either as a KL penalty to a frozen supervised reference policy or as behavior cloning on the offline dataset—the resulting guided PPO reaches the same operating point as standard PPO with roughly half the online environment interactions. Concretely, at 1M environment steps both guided variants show higher in-distribution and out-of-distribution success than standard PPO at the same budget, and RefKL matches the average OOD success of the 2M-step PPO checkpoint. The authors interpret this as evidence that offline supervision supplies a strong behavioral prior that stabilizes early optimization, after which a cu

Load-bearing premise

The paper's 'half the training budget' claim holds only if the budget is measured in environment interactions; the reference-guided variant requires extra forward passes through the frozen reference policy on every minibatch, so under wall-clock or FLOP accounting the savings may shrink or disappear.

Editorial extensions

If this is right

At equal 1M-step budgets, both RefKL and DataBC PPO outperform standard PPO on in-distribution and out-of-distribution success.
RefKL PPO at 1M steps matches the average OOD success of standard PPO at 2M steps, supporting the 'half the online budget' claim under the environment-step measure.
Stronger initial guidance (higher beta) improves both IND and OOD outcomes; the constant-beta schedule adds compute without measurable benefit.
Initializing PPO from the supervised adapter is weaker than both guided variants at the same budget, suggesting the auxiliary loss, not the initialization, is what accelerates training.
The best configuration uses strong initial guidance with a curriculum that removes the auxiliary term later, allowing the policy to finish as pure RL.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the efficiency gain is measured in environment steps, the method is most valuable where real-robot interactions dominate the cost; in cheap simulation the extra forward passes through the reference policy may erode the savings.
The reference-policy regularizer is essentially a soft distillation from a frozen policy, so a wider class of offline-guided RL objectives could be built from any pretrained policy or ensemble.
The curriculum schedule implies the auxiliary term matters most early on; testing even earlier removal or an adaptive schedule driven by policy improvement could reveal whether the 300k-step annealing is near-optimal.
The finding that supervised-initialized RL underperforms guided RL hints that fitting the supervised distribution too hard before RL can hinder later adaptation, which may generalize across parameter-efficient fine-tuning regimes.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does something useful: it tests whether two simple offline-guidance regularizers (a KL penalty to a frozen SFT reference, and a behavior-cloning loss on the offline set) make PPO fine-tuning of a LoRA-based OpenVLA policy more efficient without killing the OOD gains that pure RL provides. The answer, at the level of environment interactions, is yes. At 1M steps RefKL matches the 2M-step PPO checkpoint on OOD success (0.77 vs. 0.77) and DataBC is close (0.74); both beat 1M-step PPO (0.64). The equal-budget Table 2 is the right comparison and it supports the central claim at point-estimate level. The paper also credits what it builds on — the objectives are explicitly identified as PPD/RLHF-style KL penalties and PPO+BC — and it is unusually candid about its own limitations, flagging the small number of training seeds and the single benchmark. That honesty is earned, not perfunctory.

The new piece is the empirical transfer: these standard objectives behave differently when the policy is a frozen VLA backbone plus a small LoRA adapter. The observation that SFT-initialized PPO adapts slowly, especially on action OOD, is genuinely non-obvious and gives the paper a reason to exist beyond just reporting another win for hybrid training.

Now the soft spots, in proportion. The headline 'half the training budget' is an overstatement if budget means wall-clock or GPU time. RefKL adds a full forward pass through the reference policy for every minibatch, Section 5.5 admits this overhead is material, and no compute accounting is provided. The exact match rests on RefKL; DataBC at 1M is still slightly below 2M PPO. Also, the reference SFT checkpoint is an early-stopped 2k-demo run that the authors say is stronger than the published SFT model, but there is no stated selection protocol — that smells like a place where tuning could nudge numbers. And there are no error bars: one to two seeds, some single runs. These are addressable, not load-bearing. The environment-step claim survives scrutiny.

The paper is for people who fine-tune large VLAs for robotics and want to know whether KL-regularized PPO is worth the extra compute and bookkeeping. That audience should read it. It deserves a serious referee — likely a workshop-level but rigorous one — who can push the authors to scope the budget claim, report variance, and describe how the reference checkpoint was chosen. I would engage with it.

Referee Report

3 major / 5 minor

Summary. The paper studies whether offline supervision can be incorporated into PPO-based RL fine-tuning of large vision-language-action (VLA) models to improve training efficiency without sacrificing the out-of-distribution (OOD) generalization advantages of RL. Two guided variants are proposed: RefKL, which adds a KL penalty to a frozen SFT reference policy (Eqs. 4–5), and DataBC, which augments PPO with a behavior-cloning loss on the offline dataset (Eqs. 6–7), both with a curriculum schedule for the guidance coefficient (Eq. 8). Experiments on the RL4VLA benchmark with OpenVLA and LoRA adaptation show that, at 1M environment steps, both guided variants outperform standard PPO at the same budget, and RefKL matches the OOD success of a 2M-step PPO run (0.77 vs. 0.77). The paper concludes that offline guidance acts as an optimization prior that accelerates the early phase of RL while preserving OOD robustness.

Significance. The question is timely and practically relevant: large-scale VLA RL is sample-expensive, and any method that halves the online budget while retaining OOD generalization would be useful. The paper's strengths include the use of an established external benchmark (RL4VLA), a controlled LoRA-based OpenVLA setup, simple and clearly specified objectives, a useful ablation of the guidance strength, and an honest limitations section. The equal-budget comparison in Table 2 is internally consistent and suggestive. If the efficiency claim is confirmed with proper variance reporting and a clearly stated resource axis, the hybrid training recipe would be a meaningful empirical contribution.

major comments (3)

[Abstract, §6.1, Table 1] The headline claim 'roughly half of the training budget' is only supported for environment interactions, not for total compute. RefKL's objective (Eq. 5) requires an additional forward pass through the frozen reference policy (Eq. 4) for every minibatch, and §5.5 states that constant-β schedules add 'additional forward passes...unnecessary computational overhead.' No wall-clock, GPU-hour, or FLOP accounting is reported anywhere. If 'budget' means compute, the 1M-step RefKL vs. 2M-step PPO comparison in Table 1 may not correspond to half the budget; if it means environment steps, that axis must be stated explicitly in the abstract and §6.1. This is load-bearing for the paper's central efficiency claim.
[§5.3, Tables 2–3] All reported success rates are point estimates. Section 5.3 says experiments use 1–2 training seeds and 3 evaluation seeds, with some configurations from a single run, yet no standard errors, per-seed values, or confidence intervals are provided. The central quantitative claims—RefKL at 1M matching PPO at 2M (0.77 vs. 0.77) and the deltas in Table 2 (e.g., +0.18 vs. PPO for RefKL IND)—could be within run-to-run variance. Without variance reporting, the 'match' and even the ranking of methods is not firmly established. Please report per-seed results and error bars for all configurations.
[§5.2] The reference SFT checkpoint is an early-stopped 2k-demo, 7.5k-step model that 'achieves stronger performance than the published SFT result' and is used as the reference for RefKL and DataBC. No protocol is given for selecting this checkpoint (e.g., a fixed schedule independent of evaluation vs. selection based on validation performance). If the checkpoint was chosen because it evaluated well on the test tasks, the reference policy is optimistically biased and the guided variants' gains are inflated. State the checkpoint-selection rule and, ideally, report sensitivity to the SFT checkpoint choice.

minor comments (5)

[Tables 1–2] The column headers are ambiguous: 'IND OOD' spans five columns (Act, Lang, Vis, Avg). Please split into explicit IND and OOD subheadings so readers can map columns to categories.
[Eq. (4)] Specify the KL divergence direction (π_ref ∥ π_θ or π_θ ∥ π_ref) and note explicitly that π_ref is frozen. Also define |B| as the minibatch size.
[§4.5] The schedule (Eq. 8) uses 'training step' t; clarify whether this is an optimizer step or environment step, since the paper otherwise reports budgets in environment steps.
[Fig. 5] Indicate whether the curves are means over training seeds and add error bars, or state that only single runs are shown.
[§5.3] The sentence 'Reported success rates are averaged across both evaluation seeds and training seeds' conflicts with the preceding sentence that some configurations are reported from a single run. Clarify which configurations have 1 vs. 2 seeds.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the guided objectives are standard PPO plus an auxiliary supervised term, and the reported successes are measured independently on the external RL4VLA benchmark.

full rationale

The paper's central claims are empirical comparisons of training objectives. RefKL (Eqs. 4–5) is PPO plus a KL penalty toward a frozen SFT reference policy, and DataBC (Eqs. 6–7) is PPO plus behavior cloning on the offline dataset; these are defined independently of the reported evaluation metrics. The headline result—that RefKL at 1M environment steps matches the 2M-step PPO checkpoint on average OOD success—is a measured outcome from the external RL4VLA benchmark, not something forced by the definitions of the objectives. Hyperparameters such as β, t1, and t2 are tuned, but the success rates in Tables 1–3 are not defined in terms of those parameters, so there is no fitted-input-called-prediction pattern. The paper explicitly reproduces the RL4VLA and SFT baselines rather than importing prior fitted numbers, and it does not invoke any self-citation as a load-bearing justification or uniqueness argument. The only notable weakness is the interpretation of 'training budget' as environment interactions while RefKL adds an extra reference-model forward pass per minibatch; that is a measurement/efficiency concern, not a circularity concern. No construction-level circularity is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper contributes no new theory; its burden is hyperparameter choices (beta, schedule, reference-SFT selection) and trust in the RL4VLA/OpenVLA setup. All claims are empirical comparisons on one benchmark.

free parameters (3)

beta (offline-guidance coefficient) = 0.6 (ablated over 0.1, 0.3, 0.6)
Strength of the KL/BC auxiliary loss (Eqs. 5, 7). Central results use beta0=0.6, which Table 3 shows dominates lower values; the conclusion depends on this choice.
Curriculum endpoints t1, t2 = t1=100,000, t2=300,000
Anneal schedule in Eq. 8 chosen by hand; only a no-curriculum constant-beta comparison is reported, not a sweep over t1/t2.
Reference SFT training budget = 2k demonstrations, 7.5k steps
The reference model and SFT-init baseline use this early-stopped checkpoint, which is reported stronger than the published 16k/60k SFT result; no protocol for selecting it is given, so it acts as a tuned configuration.

assumptions (5)

domain assumption RL4VLA's modified Simpler environments and its OOD split (vision/language/action shifts) are a valid proxy for VLA out-of-distribution generalization.
Imported as the benchmark from Liu et al. (2025), Section 5.2; the paper's OOD claim is defined entirely by this benchmark.
domain assumption The sparse difference-based task-progress reward (Section 3) provides a sufficient learning signal for PPO in this setting.
Adopted without modification from RL4VLA; if the reward is poorly shaped, the reported efficiency comparison may not transfer to other rewards.
domain assumption A zero-initialized LoRA adapter on frozen OpenVLA-warmup, plus the RL4VLA value-head, is a representative parameter-efficient VLA RL configuration.
Taken from prior work (Kim et al. 2025; Liu et al. 2025); single model family limits generalization of the claims.
domain assumption KL divergence between the reference and current policy over minibatch observations (Eq. 4) is a well-defined behavior regularizer for the tokenized action distribution.
The paper does not analyze whether token-level KL corresponds to action-level closeness; this is assumed.
domain assumption Success rates from 64 IND / 960 OOD episodes and 3 evaluation seeds are stable enough to support point estimates, despite 1-2 training seeds.
Section 5.3 defines the protocol; Section 6.4 concedes variance is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models." pith.science (2026). https://pith.science/paper/IX3GR6IV

@misc{pith2026260719399,
  author       = {Pith},
  title        = {Pith review of: Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IX3GR6IV}},
  note         = {Machine review of arXiv:2607.19399}
}

read the original abstract

It is commonly observed that online reinforcement learning (RL) produces better-performing strategies than offline methods across a broad range of performance measures. In particular, RL-trained policies exhibit stronger out-of-distribution (OOD) behavior, where models trained only with imitation learning approaches often struggle. A recent study introduced an OOD-focused benchmark and reported that RL-trained vision-language-action (VLA) policies achieve noticeably better OOD performance and slightly better in-distribution (IND) performance than their counterparts trained with supervised fine-tuning (SFT). In this work, we investigate whether hybrid offline-online training can combine the advantages of both approaches. Specifically, we study RL methods regularized by offline supervision via either offline data or an offline-trained reference policy. We evaluate these approaches on the OOD benchmark and compare them with both offline-only training and standard RL. Our results show that although offline training achieves limited OOD performance by itself, incorporating offline supervision into RL preserves strong OOD capability while substantially improving training efficiency. In particular, the guided methods reach performance close to that of standard RL while requiring roughly half of the training budget. Rather than producing a trade-off between speed and OOD performance, the hybrid approach retains strong OOD capability while achieving this efficiency gain. Project page: https://alstar8.github.io/offline-supervision-vla-rl

Figures

Figures reproduced from arXiv: 2607.19399 by the authors.

**Figure 1.** The core idea of combining offline and online training in the described setting. 1. Introduction Machine learning (ML) policies can be trained either offline, using fixed datasets, or online, through interaction with the environment. In practice, online reinforcement learning (RL) often produces stronger final policies, especially when robustness and adaptation are important. At the same time, offline training remai… view at source ↗

**Figure 2.** Architecture of the OpenVLA model used in our setting. The architecture is consistent with the base OpenVLA model and the value-head modification. 2. Related Work 2.1. Reinforcement Learning for Vision–Language–Action Models Recent work has begun to study RL as a means of improving the generalization of VLA policies beyond what is obtained from SFT alone. In particular, (Liu et al., 2025) provides evidence that PPO-… view at source ↗

**Figure 3.** Training pipelines for PPO modifications used in our experiments Here Aˆ t denotes the advantage estimate and ϵ is the PPO clipping parameter. 4.3. Supervised Fine-Tuning Guided PPO For our SFT-guided RL, we supplement the PPO objective with an auxiliary loss term. We evaluate two main ways of doing this: via the frozen reference model (PPO with a KL penalty to the reference model, denoted RefKL) and with behavior c… view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Training curves for in-distribution and out-of-distribution success. Dashed lines indicate the final PPO performance at 2M steps. We use the difference-based sparse reward formulation defined in RL4VLA (Liu et al., 2025). For SFT, the expert demonstration dataset is f…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 4 linked inside Pith

[1]

Liu and F

J. Liu and F. Gao and B. Wei and X. Chen and Q. Liao and Y. Wu and C. Yu and Y. Wang , title =. Advances in Neural Information Processing Systems , year =
[2]

Nair and M

A. Nair and M. Dalal and A. Gupta and S. Levine , title =. arXiv preprint arXiv:2006.09359 , year =

arXiv 2006
[3]

Nakamoto and Y

M. Nakamoto and Y. Zhai and A. Singh and M. Sobol Mark and Y. Ma and C. Finn and A. Kumar and S. Levine , title =. Advances in Neural Information Processing Systems , volume =
[4]

A. J. Hancock and X. Wu and L. Zha and O. Russakovsky and A. Majumdar , title =. arXiv preprint arXiv:2509.22195 , year =

arXiv
[5]

Schmitt and J

S. Schmitt and J. J. Hudson and A. Zidek and S. Osindero and C. Doersch and W. M. Czarnecki and J. Z. Leibo and H. Kuttler and A. Zisserman and K. Simonyan and S. M. A. Eslami , title =. arXiv preprint arXiv:1803.03835 , year =

arXiv
[6]

Spigler , title =

G. Spigler , title =. Transactions on Machine Learning Research , year =
[7]

Ouyang and J

L. Ouyang and J. Wu and X. Jiang and D. Almeida and C. L. Wainwright and P. Mishkin and C. Zhang and S. Agarwal and K. Slama and A. Ray and J. Schulman and J. Hilton and F. Kelton and L. Miller and M. Simens and A. Askell and P. Welinder and P. Christiano and J. Leike and R. Lowe , title =. Advances in Neural Information Processing Systems , volume =
[8]

Baker and I

B. Baker and I. Akkaya and P. Zhokhov and J. Huizinga and J. Tang and A. Ecoffet and B. Houghton and R. Sampedro and J. Clune , title =. Advances in Neural Information Processing Systems , volume =

Show all 22 references

[9]

E. J. Hu and Y. Shen and P. Wallis and Z. Allen-Zhu and Y. Li and S. Wang and L. Wang and W. Chen , title =. International Conference on Learning Representations , year =
[10]

M. J. Kim and K. Pertsch and S. Karamcheti and T. Xiao and A. Balakrishna and S. Nair and R. Rafailov and E. Foster and G. Lam and P. Sanketi and Q. Vuong and T. Kollar and B. Burchfiel and R. Tedrake and D. Sadigh and S. Levine and P. Liang and C. Finn , title =. Proceedings ...
[11]

Schulman and F

J. Schulman and F. Wolski and P. Dhariwal and A. Radford and O. Klimov , title =. arXiv preprint arXiv:1707.06347 , year =

arXiv
[12]

B. D. Argall and S. Chernova and M. Veloso and B. Browning , title =. Robotics and Autonomous Systems , volume =
[13]

Ross and G

S. Ross and G. Gordon and J. A. Bagnell , title =. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics , pages =
[14]

Ren and A

Y. Ren and A. Pacchiano and P. L. Bartlett and L. Jordan and A. Z. Pan , title =. Proceedings of the 5th Conference on Robot Learning , pages =
[15]

Lee and Y

S. Lee and Y. Seo and K. Lee and P. Abbeel and J. Shin , title =. Proceedings of the 6th Conference on Robot Learning , pages =
[16]

Kang and Z

B. Kang and Z. Jie and J. Feng , title =. Proceedings of the 35th International Conference on Machine Learning , pages =
[17]

Kirkpatrick and R

J. Kirkpatrick and R. Pascanu and N. Rabinowitz and J. Veness and G. Desjardins and A. A. Rusu and K. Milan and J. Quan and T. Ramalho and A. Grabska-Barwinska and D. Hassabis and C. Clopath and D. Kumaran and R. Hadsell , title =. Proceedings of the National Academy of Scienc...
[18]

Brohan and N

A. Brohan and N. Brown and J. Carbajal and Y. Chebotar and X. Chen and K. Choromanski and T. Ding and D. Driess and P. Florence and C. Fu and others , title =. arXiv preprint arXiv:2212.06817 , year =

arXiv
[19]

Zitkovich and T

B. Zitkovich and T. Yu and S. Singh and P. Xu and J. Xiao and F. Xia and D. Driess and A. Brohan and N. Brown and C. M. de L. M. Ruiz and others , title =. Proceedings of the 7th Conference on Robot Learning , year =
[20]

Octo Model Team and T. O. Xiao and O. Padalkar and K. Pertsch and H. Huang and S. Nair and A. Bhat and K.-H. Lee and P. Liang and C. Finn , title =. International Conference on Learning Representations , year =
[21]

Black and N

K. Black and N. Jao and M. R. A. Tian and E. Ferigo and N. Duan and A. Mahler and M. Ahn and P. Bahl and C. Finn and R. E. Wang and others , title =. Robotics: Science and Systems , year =
[22]

Finn and K

Physical Intelligence and C. Finn and K. Black and N. Duan and T. Kollar and M. Ahn and collaborators , title =. Proceedings of the 9th Conference on Robot Learning , year =

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Liu and F

J. Liu and F. Gao and B. Wei and X. Chen and Q. Liao and Y. Wu and C. Yu and Y. Wang , title =. Advances in Neural Information Processing Systems , year =

[2] [2]

Nair and M

A. Nair and M. Dalal and A. Gupta and S. Levine , title =. arXiv preprint arXiv:2006.09359 , year =

arXiv 2006

[3] [3]

Nakamoto and Y

M. Nakamoto and Y. Zhai and A. Singh and M. Sobol Mark and Y. Ma and C. Finn and A. Kumar and S. Levine , title =. Advances in Neural Information Processing Systems , volume =

[4] [4]

A. J. Hancock and X. Wu and L. Zha and O. Russakovsky and A. Majumdar , title =. arXiv preprint arXiv:2509.22195 , year =

arXiv

[5] [5]

Schmitt and J

S. Schmitt and J. J. Hudson and A. Zidek and S. Osindero and C. Doersch and W. M. Czarnecki and J. Z. Leibo and H. Kuttler and A. Zisserman and K. Simonyan and S. M. A. Eslami , title =. arXiv preprint arXiv:1803.03835 , year =

arXiv

[6] [6]

Spigler , title =

G. Spigler , title =. Transactions on Machine Learning Research , year =

[7] [7]

Ouyang and J

L. Ouyang and J. Wu and X. Jiang and D. Almeida and C. L. Wainwright and P. Mishkin and C. Zhang and S. Agarwal and K. Slama and A. Ray and J. Schulman and J. Hilton and F. Kelton and L. Miller and M. Simens and A. Askell and P. Welinder and P. Christiano and J. Leike and R. Lowe , title =. Advances in Neural Information Processing Systems , volume =

[8] [8]

Baker and I

B. Baker and I. Akkaya and P. Zhokhov and J. Huizinga and J. Tang and A. Ecoffet and B. Houghton and R. Sampedro and J. Clune , title =. Advances in Neural Information Processing Systems , volume =