Pith.

Robotics · arXiv:2609.36872 v1 · submitted 29 Sep 2026 · 29 references

PreferenceFlow: Test-Time Guidance of Flow-Matching Robot Policies from Human Interventions

Yiqi Tang, Diyuan Shi, Runze Li, Donglin Wang

The paper says

“Flow-matching policies can represent complex robot behaviors but remain susceptible to local errors under distribution shift at deployment. Many reinforcement learning approaches to policy improvement require reward signals that are difficult to specify or obtain in real-world manipulation.”

more of the abstract

We present PreferenceFlow, a framework for improving a pretrained flow policy at test time without environment rewards or updates to the base policy. Human intervention chunks are paired with robot chunks generated from the same initial conditioning state to train a preference model. During infer- ence, we adopt the QGF sampling update, replacing its value gradient with the preference gradient evaluated at an estimated clean action. A gradient-cap loss penalizes excessive gradients on intervention pairs, while a zero-gradient loss discourages guidance near actions from expert demonstrations. On four real-world precision insertion tasks with a Franka robot, Pref- erenceFlow achieves a mean success rate of 90.5%, compared with 69% for the frozen policy. Ablations support the roles of gradient regularization and correctly ordered preference labels in the evaluated settings. These results demonstrate the utility of human interventions as local preference supervision for guiding frozen generative robot policies.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 30 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Hardware result is plausible and genuinely useful, but the paper's own ablation shows the preference gradient alone buys nothing — the regularizers carry the gain, and the advertised mechanism is not established. This line is wrong

4 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 29 references say.

The paper says, at Section V.E, Table II

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The preference-only variant, with its guidance weight adjusted until deployment is stable, achieves exactly 69.0% mean success — numerically identical to the frozen-policy mean. This is load-bearing for the paper's central claim (b8, b208) that human interventions provide usable preference guidance: under the reported data, the learned preference gradient contributes nothing measurable beyond the frozen sampler once guidance is forced into a deployable regime, and all 21.5pp of improvement appears only when the gradient-cap and zero-gradient regularizers are added.

more of objection 1

The paper frames this as support for the regularizers (b172), which it is, but it does not address the stronger implication that the preference content itself is not evidenced to drive the deployment gain. Please report the adjusted weights and the stability criterion used, and either provide an explanation for the 69.0% coincidence (e.g., whether the weight reduction needed for stability renders guidance negligible) or temper the mechanistic claim in the abstract and conclusion accordingly.

The paper says, at Tables I-IV, Section V

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

No confidence intervals or significance tests are reported anywhere, despite binary outcomes with n=50 trials per task. With n=50, a two-sided 95% Wilson interval for 90% success is roughly ±8pp, so per-task differences of 16-24pp are not uniformly at the same significance level.

more of objection 2

Please report binomial CIs (or a McNemar-type test where trials are paired) for the main comparison and for the ablation rows, since the 21.5pp headline gain and the 69.0%-vs-69.0% ablation claim both depend on effect sizes that are small relative to per-task uncertainty.

The paper says, at Section III and b204

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.86):

“Human actions are collected under closed-loop control, whereas each robot candidate is generated from the initial observation; longer windows may therefore yield noisier comparisons despite shared initial con- ditioning.”

Pith says: Objection 3 of 4 · major

The supervision assumption is that a human chunk is preferred to a robot chunk regenerated from the same window-start state (b203). As the authors note (b204), the human chunk is collected under closed-loop execution while the robot candidate is generated from a single initial observation, so the learned preference may encode the value of closed-loop feedback rather than intrinsic action quality.

more of objection 3

This is acknowledged as a limitation, but it directly affects the interpretation of the headline result on the same task distribution. A concrete control would strengthen the paper: for a subset of trials, execute the regenerated robot candidate closed-loop and compare the preference model's ranking against that outcome, or compare against guidance with a preference model trained on pairs where both candidates are closed-loop rollouts.

The paper says, at Section V.E, adjusted-guidance protocol

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The 'adjusted guidance' protocol — progressively reducing each ablated variant's weight until deployment is stable — is a post hoc selection rule applied asymmetrically (the full model uses its own weight, while each ablated variant is tuned). As reported, this makes the 81.0%, 74.5%, and 69.0% adjusted figures informally comparable at best. Please specify the stability criterion (what constitutes 'unstable'), the selected weight for each variant, and whether the conclusion is robust to the choice of anchor weight.

5 minor objections

The paper says, at Section IV.C, Eq. (12)

Pith says: minor objection 1 of 5

The relation between the KL motivation (Eq. 10) and the actual update (Eq. 12) is loose, as the paper acknowledges (b120); consider stating explicitly in the main text that Eq. (12) is a heuristic application of the QGF update and is not derived from Eq. (10), rather than leaving this to the discussion.

The paper says, at Section V.B, Eq. (7)

Pith says: minor objection 2 of 5

The noise scale σ_a in the zero-gradient loss is not specified; its value affects the anchor neighborhood and should be reported.

show 3 more

The paper says, at Section II.A

Pith says: minor objection 3 of 5

Minor typo: 'World-action models (W AMs)' — the spacing should be 'WAMs'.

The paper says, at Figures 1 and 2

Pith says: minor objection 4 of 5

The figure labels 'π₀.₅vla' and the schematic arrow '∇â1Fφ(s,â1)' should be typeset consistently with the body notation and with Eq. (12).

The paper says, at Section V.D

Pith says: minor objection 5 of 5

The statement that the gains are 24, 22, 24, and 16pp on the four tasks would benefit from per-task counts (e.g., 45/50 vs 33/50) to make the n=50 scale transparent.

What would settle it

On held-out intervention windows not used in training, measure whether preference-only guidance shifts open-loop robot chunks toward the human chunks that define the preference direction: for a sample of held-out states s, draw a robot chunk from the frozen policy, run the Eq. 12 update with the preference model and no regularizers at several stable guidance weights, and compare L2 distance to the held-out human chunk against the unguided sample. If guided chunks are not systematically closer to human actions on held-out states, the preference gradient does not encode usable local action quality and the deployment gains must be attributed to the regularizers; if they are closer, the mechanism survives and the 69.0% result is merely a scale-tuning artifact of the preference-only configuration.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 30 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 44 s · 11,832 tokens read, 5,821 written
  3. Editor wrote the verdict above 32 s · 12,907 tokens read, 4,317 written
  4. Referee major revision: 4 major objections, 5 minor 33 s · 12,281 tokens read, 4,215 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Two things to know. The hardware result is real and plausible: 90.5% vs 69% over 200 trials, gains in the same direction on all four insertion tasks, and a clean monotone curve as intervention episodes go 10 to 50. The mechanism it advertises is not established by its own ablations. The preference-only variant, tuned down to a stable guidance weight, lands at exactly 69.0% — the frozen baseline mean — so all 21.5 points come from the two hand-designed regularizers, not from the preference gradient direction the paper holds up as the headline (b175, Table II).

What is actually new: the synthesis. QGF's flow-guidance update, DPO's exponential tilt, and the DAgger family are each prior art, but nobody has trained a state-conditioned preference model from human-robot chunk pairs sharing a window-start state and fed its gradient into QGF with these two regularizers. That combination is a legit contribution, and the paper is refreshingly honest: it flags the closed-loop/open-loop label confound (human actions see real-time feedback the robot generator never sees), the reliance on the base policy reaching correctable states, and the fact that removing the zero-gradient loss also removes demonstration anchors (b204, b205, b168).

Soft spots, in proportion. First and load-bearing: the 69% preference-only ablation. Framing it as support for the regularizers is fair, but it cuts against the "human interventions as local preference supervision" claim. One nuance worth keeping: the score itself does carry signal, since preference reranking hits 92% and randomized labels fall to 63.5% (Table IV). So it is specifically the gradient-guidance mechanism, not the preference model, that lacks attribution. Also, the adjusted-weight protocol tunes each variant's weight post hoc, so the comparison is not at matched guidance strength; you cannot rule out that the pure preference signal was simply run too weak. That is the charitable reading, and the paper does not offer it. Second, there are no confidence intervals or significance tests on any table. 181 vs 138 successes in 200 is almost certainly real, but error bars are the field norm and cheap to add. The confounded zero-gradient ablation is disclosed, so minor.

Who it is for: anyone in test-time adaptation of frozen VLAs. The positive result is a useful data point, and the ablation carries a real lesson — gradient guidance of a preference head goes nowhere until you shape the gradient field.

Recommendation: send it to review. A serious referee should push on Table II and make the authors either narrow the mechanism claim or add a matched-strength control. Not a reject.

The strongest objection

The central claim is that human interventions supply usable local preference supervision that guides a frozen flow policy (b8, b208). The strongest direct evidence for the mechanism is the ablation that removes both gradient regularizers, keeping only the preference score and its downstream gradient guidance. With the guidance weight adjusted until deployment is stable, this preference-only variant achieves exactly 69.0% mean success (Table II, b175), numerically identical to the frozen baseline's 69.0%. At the full model's weight it is unstable (Table II). So once guidance is forced into a deployable regime, the learned preference gradient contributes nothing measurable beyond the frozen sampler; all 21.5pp of improvement materializes only when the two hand-designed gradient-shape regularizers (gradient-cap on intervention pairs, zero-gradient near demonstrations, Eqs. 6-7) are added. The randomized-label control (63.5%, Table IV, b200) shows that label ordering matters, and the architecture ablation (Table III) shows the score head matters, but both results are reported in the presence of the regularizers; neither demonstrates that the preference gradient direction itself carries the deployment gain. The paper frames the preference-only result positively, as support for the regularizers (b172), rather than as evidence that the Eq. 12 guidance update contributes nothing. This is not an acknowledged limitation: the Limitations section discusses the closed-loop/open-loop label confound (b204) and the reliance on the base policy reaching correctable states (b205), but never flags that the tuned preference signal alone yields zero improvement. What is actually evidenced is that a regularized gradient-shaping procedure built on intervention data improves success; the claim that preference content, rather than the regularization geometry, drives guidance (the 'local preference supervision' headline) is not yet established.

I read the paper in good faith and agree the empirical headline is solid: 181 vs 138 successes over 200 real-robot trials (b150), consistent across all four tasks, with a negative control that loses to the frozen policy. Those are genuine directional results. The reader's chosen weakest assumption, the closed-loop-versus-open-loop label confound (b204), is real but is explicitly acknowledged by the authors in the Limitations section, and I find it less damaging than they do: at deployment the robot also executes open-loop K-step chunks with replanning, so guiding those open-loop chunks toward actions that succeeded under closed loop is arguably the desired target behavior, and the confound does not obviously inflate the measured success. My identified concern is different and, in my reading, not acknowledged: the preference-only ablation lands at precisely the frozen mean (69.0% = 69.0%, Table II, b175), indicating that after weight tuning the preference gradient contributes zero observable guidance. That directly undercuts the mechanistic claim that human preference content drives the improvement (b8, b208); the gain is attributable instead to the interaction of the preference model with the two hand-designed gradient regularizers. This does not overturn the empirical full-method result, which is measured directly on the robot, so I keep the reader's CONDITIONAL verdict rather than moving to REJECT; however, it strengthens the case for conditionality because the central mechanism attributed to preference learning is not evidenced by the internal ablations. A held-out alignment test (does preference-only guidance move sampled chunks toward held-out human chunks?) would settle whether the preference gradient carries real signal or the improvement is pure regularizer geometry. I rate agreement with the reader's weakest assumption as partial: same general region of the argument, different specific ins

The referee's report

If the central claim holds, the paper contributes a practical recipe for adapting a frozen generative policy with human intervention data: 200 real-robot trials, consistent gains across all four tasks, and a full method that beats Intervention BC, QGF, and FlowDAgger. The paper deserves credit for transparent ablation reporting, for limiting its claims about Eq. (10) to motivation (b110, b120), and for acknowledging several scope conditions (b162, b191, b202-b207). The single most important reservation is that the preference-only ablation (Table II) lands at exactly the frozen-policy mean of 69.0%, which undercuts the attribution of the measured gain to preference content as opposed to the two hand-designed gradient regularizers; this point is reported but its implication is not confronted. A second reservation is the closed-loop/open-loop asymmetry in the supervision pairs, which the authors acknowledge (b204) but which bears directly on what the preference model learns.

Pith's reading of the paper (a machine's summary, not the authors')

PreferenceFlow claims that a few seconds of human intervention during robot execution can serve as local preference supervision for improving a frozen flow-matching policy at test time, with no environment rewards and no update to the base policy. The core idea is to pair each human action chunk with a robot chunk regenerated from the same window-start state, train a state-conditioned preference model on those pairs, and then steer the frozen flow sampler by adding the preference gradient during denoising. On four real-world precision-insertion tasks with a Franka robot, the method reports a mean success rate of 90.5% versus 69% for the frozen policy, a gain of 21.5 percentage points over 200 trials, and it outperforms fine-tuning on interventions (84.5%). The claim, if correct, is that human corrections need not be replayed or used to retrain a policy: they can be distilled into a lightweight guidance signal applied only at sampling time.

What carries the argument. The central object is the state-conditioned preference score $F_\phi(s, a)$, a scalar built from the frozen policy's conditioning tokens (visual, language, and robot-state) via per-step cross-attention over independently scored action tokens. The mechanism is QGF's guided Euler update with the Q-function replaced by the preference score: each noisy action is mapped to an estimated clean endpoint $\hat a_1 = a_t + (1-t)v_\theta(s, a_t, t)$, and the sampler follows $a_{t+\delta} = a_t + \delta\big(v_\theta(s,a_t,t) + \lambda_{\text{guide}}\nabla_{\hat a_1}F_\phi(s, \hat a_1)\big)$. Two gradient regularizers keep the guidance corrective but bounded: a gradient-cap loss that penalizes preference gradients above a threshold on intervention pairs, and a zero-gradient loss that penalizes gradients near demonstrated actions. Everything operates on a frozen flow backbone; only the preference-model parameters are updated during adaptation.

Core claim. The paper establishes that an intervention-trained preference score can replace the Q-function inside QGF's flow-guidance update and meaningfully improve a frozen pretrained flow policy. From human-robot action-chunk pairs sharing a window-start state, a state-conditioned preference model is trained with a Bradley-Terry objective, motivated by the exponential-tilting argument of KL-regularized preference optimization. At inference, each noisy action is extrapolated to an estimated clean endpoint, the preference gradient there is added to the frozen velocity field, and two regularizers constrain the guidance: a gradient-cap loss that limits excessive gradients on intervention pairs, and a zero-gradient loss that suppresses guidance near demonstrated routine actions. The central empirical claim is that this guidance lifts mean success on four real precision-insertion tasks from 69% to 90.5%, with ablations supporting the regularizers, independent per-step action scoring, and correctly ordered preference labels.

The premise it rests on. The load-bearing premise is that a human action chunk is strictly preferable to a robot chunk regenerated from the same window-start state, but human actions are collected under closed-loop feedback that the robot generation never sees, so the learned preference may reward closed-loop information rather than intrinsic action quality.

If the paper is right (Pith's inferences, not the authors')

Human interventions can be converted into local preference supervision that improves a frozen generative policy without environment rewards or base-policy parameter updates.

Mean success on four precision-insertion tasks rises from 69% (frozen policy) to 90.5%, a 21.5 percentage-point gain over 200 trials, exceeding Intervention BC (84.5%), QGF (76%), and FlowDAgger (78.5%).

The gradient regularizers are load-bearing: removing either the gradient-cap or zero-gradient loss degrades mean success to 15% or 5% at fixed guidance, and to 81% or 74.5% even after tuning the guidance weight down; preference-only guidance collapses to the frozen-policy level.

The learned score also supports sample-based reranking, selecting among five candidates to reach 92%, while randomized preference labels drop success to 63.5%, below the frozen policy, so label ordering matters.

Supervision is comparatively data-efficient in the tested range: 10 intervention episodes already add 9.5 percentage points, with diminishing but positive gains up to 50 episodes.

Because human chunks are collected under closed-loop feedback while robot candidates are generated open-loop from a single observation, the learned preference may be encoding the value of closed-loop information rather than intrinsic action quality; a direct test would pair a robot chunk generated open-loop with the same chunk re-executed under closed-loop feedback and check which the preference f

The guidance mechanism lives at the sampler level and never touches policy weights, so the same preference-gradient recipe could in principle be lifted onto other generative backbones (for instance diffusion policies) or composed with runtime cost-based steering objectives.

The architecture ablation — removing temporal self-attention between action tokens helps success rise to 90.5% from 77% — suggests the preference model is most useful when it isolates per-step corrections rather than global action style; a mechanism test would check whether per-step scores align with the actual error locations in failed rollouts.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Flow-matching policies can represent complex robot behaviors but remain susceptible to local errors under distribution shift at deployment. Many reinforcement learning approaches to policy improvement require reward signals that are difficult to specify or obtain in real-world manipulation. We present PreferenceFlow, a framework for improving a pretrained flow policy at test time without environment rewards or updates to the base policy. Human intervention chunks are paired with robot chunks generated from the same initial conditioning state to train a preference model. During infer- ence, we adopt the QGF sampling update, replacing its value gradient with the preference gradient evaluated at an estimated clean action. A gradient-cap loss penalizes excessive gradients on intervention pairs, while a zero-gradient loss discourages guidance near actions from expert demonstrations. On four real-world precision insertion tasks with a Franka robot, Pref- erenceFlow achieves a mean success rate of 90.5%, compared with 69% for the frozen policy. Ablations support the roles of gradient regularization and correctly ordered preference labels in the evaluated settings. These results demonstrate the utility of human interventions as local preference supervision for guiding frozen generative robot policies.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-30

The authors' main claim, as the classifier picked it out 1.00

“These results demonstrate the utility of human interventions as local preference supervision for guiding frozen generative robot policies.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.23 · human participants 0.73 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.96 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Flow-matching policies can represent complex robot behaviors but remain susceptible to local errors under distribution shift at deployment. background 0.96 0.09 0.02 asserted 0.99
Many reinforcement learning approaches to policy improvement require reward signals that are difficult to specify or obtain in real-world manipulation. background 0.86 0.04 0.02 asserted 0.99
We present PreferenceFlow, a framework for improving a pretrained flow policy at test time without environment rewards or updates to the base policy. contribution 1.00 0.90 0.15 asserted 1.00
Human intervention chunks are paired with robot chunks generated from the same initial conditioning state to train a preference model. method 0.99 0.91 0.03 asserted 0.99
During infer- ence, we adopt the QGF sampling update, replacing its value gradient with the preference gradient evaluated at an estimated clean action. method 1.00 0.87 0.04 asserted 1.00
A gradient-cap loss penalizes excessive gradients on intervention pairs, while a zero-gradient loss discourages guidance near actions from expert demonstrations. method 1.00 0.86 0.03 asserted 0.97
On four real-world precision insertion tasks with a Franka robot, Pref- erenceFlow achieves a mean success rate of 90.5%, compared with 69% for the frozen policy. result 1.00 0.97 0.32 asserted 1.00
Ablations support the roles of gradient regularization and correctly ordered preference labels in the evaluated settings. result 0.92 0.94 0.07 asserted 0.96
These results demonstrate the utility of human interventions as local preference supervision for guiding frozen generative robot policies. interpretation 1.00 0.89 0.90 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-30: 1 request, 6,798 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (4)
Figure 1 of the paper
Figure 1. Overview of PreferenceFlow. Human interventions provide action-chunk comparisons sharing a window-start state, while expert demonstrations
Figure 2 of the paper
Figure 2. State-conditioned preference-model architecture. Three camera
Figure 3 of the paper
Figure 3. Real-world precision insertion tasks. Each row shows one task and progresses from left to right, from the initial approach to insertion.
Figure 4 of the paper
Figure 4. Mean success (%) across four tasks as a function of intervention

Figures are the authors', reproduced from the paper.

References (29)
  1. [1] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” inProc. Robotics: Science and Systems (RSS), 2023 on Pith
  2. [2] Y . Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” inProc. Int. Conf. Learning Representations (ICLR), 2023 on Pith
  3. [3] K. Black et al., “π 0: A vision-language-action flow model for general robot control,” inProc. Robotics: Science and Systems (RSS), 2025 on Pith
  4. [4] Physical Intelligence et al., “π 0.5: A vision-language-action model with open-world generalization,”arXiv preprint arXiv:2504.16054, 2025 on PithPith's reviewarXiv
  5. [5] Z. Zhou, A. Peng, C. Xu, Q. Li, T. Springenberg, K. Frans, and S. Levine, “Test-time gradient guidance of flow policies in reinforce- ment learning,”arXiv preprint arXiv:2606.11087, 2026 on PithPith's reviewarXiv
  6. [6] L. Ouyang et al., “Training language models to follow instructions with human feedback,” inAdvances in Neural Information Processing Systems, 2022 on Pith
  7. [7] Gemini Robotics Team et al., “Gemini Robotics 1.5: Pushing the fron- tier of generalist robots with advanced embodied reasoning, thinking, and motion transfer,”arXiv preprint arXiv:2510.03342, 2025 on PithPith's reviewarXiv
  8. [8] NVIDIA et al., “GR00T N1: An open foundation model for generalist humanoid robots,”arXiv preprint arXiv:2503.14734, 2025 on PithPith's reviewarXiv
  9. [9] M. Shukor et al., “SmolVLA: A vision-language-action model for affordable and efficient robotics,”arXiv preprint arXiv:2506.01844, 2025 on PithPith's reviewarXiv
  10. [10] S. Ye et al., “World action models are zero-shot policies,”arXiv preprint arXiv:2602.15922, 2026 on PithPith's reviewarXiv
  11. [11] M. J. Kim et al., “Cosmos Policy: Fine-tuning video models for visuomotor control and planning,”arXiv preprint arXiv:2601.16163, 2026 on PithPith's reviewarXiv
  12. [12] G. Yan, J. Liu, Y . Fan, L. Cai, M. Liao, J. Zhang, and D. Fox, “Flex- π: A multi-stream world-action model with compute flexibility,”arXiv preprint arXiv:2608.10860, 2026 on PithPith's reviewarXiv
  13. [13] Y . Wang et al., “OpenW AM: An open, modular exploration to- wards systematic world-action model pretraining,”arXiv preprint arXiv:2609.07398, 2026 on PithPith's reviewarXiv
  14. [14] A. Hung, B. P. Duisterhof, D. Ramanan, and J. Ichnowski, “Modality-autoregressive world-action models,”arXiv preprint arXiv:2609.17524, 2026 on PithPith's reviewarXiv
  15. [15] Physical Intelligence et al., “π ∗ 0.6: A VLA that learns from experi- ence,”arXiv preprint arXiv:2511.14759, 2025 on PithPith's reviewarXiv
  16. [16] A. Wagenmaker, M. Nakamoto, Y . Zhang, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine, “Steering your diffusion policy with latent space reinforcement learning,”arXiv preprint arXiv:2506.15799, 2025 on PithPith's reviewarXiv
  17. [17] C. Xu, J. T. Springenberg, M. Equi, A. Amin, A. Esmail, S. Levine, and L. Ke, “RL Token: Bootstrapping online RL with vision-language- action models,”arXiv preprint arXiv:2604.23073, 2026 on PithPith's reviewarXiv
  18. [18] J. Lu, X. Qin, Y . Jiang, K. Wang, C. Zhang, B. Liang, J. Yang, M. Xu, and L. Zhao, “UniSteer: Unified noise steering for efficient human- guided VLA adaptation,”arXiv preprint arXiv:2605.10821, 2026 on PithPith's reviewarXiv
  19. [19] A. Tang, W. Chen, A. Wagenmaker, C. Finn, and S. Levine, “Im- proving robotic generalist policies via flow reversal steering,”arXiv preprint arXiv:2606.13675, 2026 on PithPith's reviewarXiv
  20. [20] M. Murray, D. Chen, S. Bagaria, D. Fortier, T. Hellebrekers, G. Mullins, H. Gajarla, O. Mees, M. Cakmak, and A. Kolobov, “FlowDAgger: Human-in-the-loop adaptation of generative robot poli- cies in latent space,”arXiv preprint arXiv:2607.08877, 2026 on PithPith's reviewarXiv
show 9 more
  1. [21] Y . Jia and J. P. How, “Steering generative robot policies with lexico- graphic preferences,”arXiv preprint arXiv:2609.15014, 2026 on PithPith's reviewarXiv
  2. [22] P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” inAdvances in Neural Information Processing Systems, 2017 on Pith
  3. [23] D. S. Brown, W. Goo, P. Nagarajan, and S. Niekum, “Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations,” inProc. Int. Conf. Machine Learning (ICML), 2019 on Pith
  4. [24] K. Lee, L. Smith, and P. Abbeel, “PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsu- pervised pre-training,” inProc. Int. Conf. Machine Learning (ICML), 2021 on Pith
  5. [25] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” inAdvances in Neural Information Processing Systems, 2023 on Pith
  6. [26] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” inProc. Int. Conf. Artificial Intelligence and Statistics (AISTATS), 2011 on Pith
  7. [27] J. Zhang and K. Cho, “Query-efficient imitation learning for end-to- end simulated driving,” inProc. AAAI Conf. Artificial Intelligence, 2017 on Pith
  8. [28] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer, “HG-DAgger: Interactive imitation learning with human experts,” in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2019 on Pith
  9. [29] J. Luo, C. Xu, J. Wu, and S. Levine, “Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning,”arXiv preprint arXiv:2410.21845, 2024 on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment