Pith. sign in

REVIEW 3 major objections 6 minor 30 references

Human corrections can adapt frozen generative robot policies by mapping each expert action into the noise that would have produced it, then training a small latent steerer.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 06:04 UTC pith:JXRU7YJY

load-bearing objection Clean, practical method for steering frozen generative robot policies from human corrections via action inversion; evidence is solid within the stated manifold-support limit. the 3 major comments →

arxiv 2607.08877 v1 pith:JXRU7YJY submitted 2026-07-09 cs.RO cs.LG

FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space

Michael Murray , Daphne Chen , Simran Bagaria , Dean Fortier , Tess Hellebrekers , Galen Mullins , Harshavardhan Gajarla , Oier Mees
show 2 more authors
Maya Cakmak Andrey Kolobov
This is my paper
classification cs.RO cs.LG
keywords generative robot policiesflow matchingdiffusion policyhuman-in-the-loopDAggerlatent-space adaptationaction inversionrobotic manipulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Pretrained flow-matching and diffusion robot policies encode strong behavioral priors, but fail on real-world edge cases outside their training mixture. Retraining or fine-tuning the whole model is slow, expensive, and often erases skills the base already had. FlowDAgger keeps the base frozen and instead converts each human corrective action into the noise vector that would have made the base policy output that action, using reverse-time integration and local refinement. Those inverted noises supervise a lightweight latent policy that chooses noise at deployment, so the same generative process is steered without changing its weights. Across simulation and real single-arm and bimanual hardware, including action-head VLAs and world-action models, a handful of interventions raises success while held-out pretrained skills stay largely intact.

Core claim

The paper claims that action inversion—recovering a noise vector w* such that a frozen generative policy maps (observation s, w*) to a human corrective action a*—turns sparse human interventions into supervision for a small noise-space policy, enabling sample- and compute-efficient adaptation of flow-matching and diffusion robot policies without weight updates and with better preservation of pretrained skills than fine-tuning or action-space residual methods.

What carries the argument

Action inversion: reverse-time per-step fixed-point integration (and a joint world-action variant for world-action models) that maps each expert action a* at observation s to noise w* with π_gp(s, w*) ≈ a*, providing targets for a lightweight noise policy that replaces the usual Gaussian draw at deployment.

Load-bearing premise

Desired corrections must already lie close enough to behaviors the frozen base can produce; if the needed skill is far outside that manifold, inversion only recovers the nearest representable base behavior and cannot invent new ones.

What would settle it

On a task where the expert correction is known to lie outside the base policy’s action manifold (for example a contact strategy never seen in pretraining), measure whether inverted noise still reconstructs the expert action closely and whether the adapted noise policy improves success; failure of both reconstruction and adaptation would falsify the method’s core claim for out-of-support skills.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. FlowDAgger adapts frozen generative robot policies (flow-matching and diffusion action heads, and world-action models) from human interventions by mapping each corrective action a* at observation s to a noise vector w* such that the frozen base map satisfies π_gp(s, w*) ≈ a* (action inversion via per-step fixed-point reverse Euler for few-step ODEs, and joint EDM inversion with action-frame minimal-delta targets for WAMs). A lightweight noise policy is then trained by supervised regression on inverted corrections plus successful autonomous noises (dual buffer), and replaces the base noise draw at deployment without updating base weights. Experiments on MetaWorld (π0.5 and Cosmos-Policy), Gr00t N1.7, a diffusion policy, and eight real single-arm/bimanual tasks report higher success under matched budgets than SFT, LoRA-DAgger, Residual-DAgger, and DSRL, with better held-out prior preservation than weight-space fine-tuning.

Significance. If the results hold, the paper offers a practical, compute-light interface for adapting robot foundation models in the real world: few human interventions, consumer-GPU training, and no weight updates that erode pretrained skills. The combination of inversion-based noise targets with DAgger-style collection is a clear methodological contribution relative to residual action-space DAgger and reward-driven latent RL (DSRL). Strengths include multi-family transfer (action-head VLAs and WAMs), explicit inversion accuracy ablations (App. B.1–B.2), prior-preservation measurements (Table 3), and real-hardware gains from 5–20 intervention episodes (Table 4). The manifold-support limit is stated honestly in §6.

major comments (3)
  1. Table 4 (real hardware) reports only Base, SFT, and FlowDAgger. The central claim that noise-space adaptation outperforms action-space residual and weight-space DAgger is supported in simulation (Table 1, Fig. 3) but not isolated on hardware, where Residual-DAgger and LoRA-DAgger are absent. Because residual methods are the closest architectural competitors and can be run with the same intervention stream, their omission leaves the real-world advantage of the latent interface under-supported. Adding at least Residual-DAgger (or a residual action-space control) under the same intervention budget would make the hardware claim load-bearing rather than suggestive.
  2. §4.2 and App. A claim that joint world-action inversion with a minimal-delta action-frame swap yields usable (s, w*) targets for Cosmos-Policy, but unlike the action-head case (App. B.1, Table 5: Action MSE and downstream SR by inverter), the manuscript does not report reconstruction error or success-rate sensitivity for the WAM inverter (joint vs. action-frame-only, with/without terminal Adam). Table 2 shows task gains, which is necessary but not sufficient to establish that inversion fidelity—not other factors—drives those gains. A short quantitative reconstruction table for Cosmos-Policy analogous to Table 5 would close this gap.
  3. §4.3 dual-buffer training (equal mix of inverted corrections and successful autonomous noises) is presented as essential to avoid overfitting sparse interventions, yet no ablation of the buffer mix or of the autonomous buffer alone appears in the main results. Given that free parameters listed for the method include this ratio, a small sensitivity study (e.g., corrections-only vs. 50/50 vs. autonomous-heavy) on one MetaWorld task would show whether the dual buffer is load-bearing for the reported sample efficiency.
minor comments (6)
  1. Fig. 3 caption and §5.1: clarify whether intervention frequency and human gating criteria are matched across FlowDAgger, Residual-DAgger, and LoRA-DAgger, or only the total rollout budget N=50.
  2. Eq. (3) and App. B.2: state the practical stopping criterion or residual tolerance used online, not only M=5 as default, so implementers can verify contraction on new base models.
  3. Table 3: held-out prior preservation is shown only for π0.5 after Hammer adaptation; a one-sentence note on whether similar checks were run for Cosmos-Policy or real tasks would help scope the claim in §5.4.
  4. App. A.3: the PCA basis rank k=64 for WAM noise is a free parameter; briefly report how k was chosen and whether full joint regression was unstable only in wall-clock or also in SR.
  5. Minor notation: π_gp(s, w) is used for the noise-to-action map while π_gp(a|s) denotes the induced distribution; a short glossary or consistent bolding of noise vs. action would reduce ambiguity in §3–4.
  6. Website link is given; if code/inversion scripts will be released, stating that in the camera-ready would strengthen reproducibility claims.

Circularity Check

0 steps flagged

No significant circularity: empirical adaptation method with external task-success metrics; inversion and noise-policy regression do not reduce claims to their inputs by construction.

full rationale

FlowDAgger’s load-bearing chain is (i) reverse-time fixed-point inversion of a frozen generative map π_gp to obtain noise targets w* for human actions a* (Eq. 3 / §4.1–4.2), (ii) supervised MSE regression of a lightweight noise policy on those targets plus successful autonomous noise (Eq. 5 / §4.3), and (iii) evaluation by external task success rates under matched intervention/episode budgets (Tables 1–4, Figs. 3–6). None of these steps is self-definitional: task success is not defined by the training loss or by the inversion residual. Inversion is a computational recovery of w* such that π_gp(s, w*) ≈ a*, not a fit that is later relabeled as a prediction. Baselines (SFT, LoRA/Residual-DAgger, DSRL) and held-out prior-preservation tests are independent external comparisons. Related-work citations (DAgger, DSRL, residual DAgger, image-diffusion inversion) supply background and contrast, not a uniqueness theorem or ansatz that forces the reported gains. The paper’s own limitation (§6)—that desired a* must lie in the support of the frozen map—is an acknowledged scope bound, not a circular reduction. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central claim rests on standard generative-policy math, the domain premise that noise is a valid steering surface, invertibility of few-step Euler maps via fixed-point iteration when Δt L < 1, and several hand-chosen training/inversion hyperparameters. No new physical entities are postulated; the method invents procedures (action inversion, dual-buffer noise policy) rather than unobserved mediators. Free parameters affect efficiency and stability but not the logical form of the claim.

free parameters (5)
  • Fixed-point iterations M = 5
    Default M=5 chosen from Action-MSE/wall-clock sweep (Table 6); inversion fidelity and downstream SR depend on this choice.
  • Dual-buffer sampling ratio = 1:1
    Equal proportion of inverted corrections and successful autonomous noises per batch is a hand-set design choice that anchors the noise policy to the base prior.
  • WAM noise PCA rank k = 64
    Joint latent ~1e5 dims reduced to k=64 PCA coefficients for the noise policy; chosen for stability, not derived.
  • Euler steps K / schedule = K~10; σ_min=4
    Base policy discretization (e.g. K≈10 for flow heads; EDM σ_max=80, σ_min=4 for Cosmos) is inherited but inversion accuracy depends on it.
  • Terminal Adam solve for EDM denoise
    WAM inversion uses a short local Adam solve at nonzero σ_min; step count/lr are implementation choices affecting reconstruction error.
axioms (5)
  • domain assumption A pretrained generative policy is a deterministic noise-to-action map via ODE/Euler integration of a learned velocity field (Eqs. 1–2).
    Standard flow-matching/diffusion policy formulation (§3); required for defining inversion.
  • domain assumption Replacing w ~ N(0,I) with a state-conditioned noise steers behavior without changing θ.
    Architectural premise shared with DSRL (§3, Noise as the adaptation surface); load-bearing for frozen-base adaptation.
  • standard math Per-step map x_k = x_{k+1} − Δt v_θ(x_k, t_k, s) is a contraction when Δt L < 1, so fixed-point iteration converges.
    Invoked to justify M=5 fixed-point inversion (Eq. 3, §4.1).
  • domain assumption Human-gated corrective actions on states visited by the deployed policy are informative supervision for reducing covariate shift.
    DAgger/HG-DAgger premise used throughout the adaptation loop (§3–4.3).
  • ad hoc to paper For WAMs, a minimal-delta swap of only the action frame onto the base clean latent stays on the base manifold and yields a valid joint inversion target.
    Construction in §4.2; not independently validated outside this paper’s Cosmos-Policy experiments.
invented entities (2)
  • Action inversion procedure (per-step fixed-point + WAM joint variant) no independent evidence
    purpose: Map expert actions a* to noise targets w* usable as supervised labels for a latent policy.
    Core methodological invention; independent evidence is empirical reconstruction MSE and downstream SR, not an external physical prediction.
  • Dual-buffer noise policy π_w no independent evidence
    purpose: Lightweight observation→noise map trained on inverted corrections plus successful autonomous noises to steer the frozen base.
    Standard MLP architecture with a paper-specific data mixture; not a new physical entity, but a new control interface for this setting.

pith-pipeline@v1.1.0-grok45 · 18674 in / 3714 out tokens · 44071 ms · 2026-07-13T06:04:28.394527+00:00 · methodology

0 comments
read the original abstract

Pretrained generative robot policies based on flow matching and diffusion have achieved impressive results across a wide range of manipulation tasks. Yet real-world deployments routinely expose failure modes outside the pretraining distribution. Closing these gaps typically requires large-scale data collection or online reinforcement learning on physical hardware, which is impractical for rapid and safe adaptation. We present FlowDAgger, a sample- and compute-efficient method for adapting frozen generative robot policies from human interventions in latent space. Our key idea is action inversion: each human expert action is mapped to the noise that would have produced it under the frozen base policy, using reverse-time integration followed by local refinement. The resulting inverted noise provides supervision for a lightweight latent policy that steers the base model at deployment time, enabling rapid skill acquisition while preserving its behavioral priors. We evaluate FlowDAgger in simulation and on real-world bimanual and single-arm manipulation, adapting both action-head VLAs and world-action models from a handful of interventions. FlowDAgger outperforms supervised fine-tuning and latent-space RL baselines and preserves pretrained skills on held-out tasks, offering a practical path for adapting robot foundation models in the real world. Website: https://microsoft.github.io/FlowDAgger

Figures

Figures reproduced from arXiv: 2607.08877 by Andrey Kolobov, Daphne Chen, Dean Fortier, Galen Mullins, Harshavardhan Gajarla, Maya Cakmak, Michael Murray, Oier Mees, Simran Bagaria, Tess Hellebrekers.

Figure 1
Figure 1. Figure 1: An overview of FLOWDAGGER. A pre-trained generative policy πgp producing actions a is deployed with a human operator in the loop. When the operator intervenes, the corrective action a ∗ is inverted back to a vector w ∗ in the policy’s latent space. The resulting (s, w∗ ) pairs supervise a lightweight latent-space policy that adapts the policy without modifying the base model’s weights. an additive correcti… view at source ↗
Figure 2
Figure 2. Figure 2: Real-world evaluation tasks. Left to right: Glassware Stacking, the BusyBox bench￾mark [29] (Slider, Button, Wire Pull), Toolbox Packing, Jenga Stacking, and Plug Insertion. 0 50 100 0.00 0.25 0.50 0.75 1.00 Success rate assembly 0 50 100 door_lock 0 50 100 hammer 0 50 100 stick_push 0 50 100 hand_insert Episodes base π0.5 DSRL Residual-DAg FlowDAgger [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Success rate vs. adaptation rollouts for FlowDAgger (green), Residual-DAgger (blue), and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Assembly success rate at N=50 vs. peak training VRAM (upper-left is better). Because FlowDAgger steers the frozen policy through its noise input, adaptation should change the task behavior with￾out rewriting unrelated skills. We test this by adapting each method on Hammer (50 episodes; 50 demos for SFT at a matched gradient-step budget) and evaluating on five held-out tasks that the base policy nearly satu… view at source ↗
Figure 5
Figure 5. Figure 5: Success rate vs. additional training episodes for FlowDAgger (green) and DSRL (purple) [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Success rate vs. additional training episodes for FlowDAgger (green) and DSRL (purple) [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 11 linked inside Pith

  1. [1]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. InProceedings of Robotics: Science and Systems (RSS), 2023

    2023

  2. [2]

    Black, N

    Physical Intelligence, K. Black, N. Brown, J. Darpinian, D. Driess, C. Finn, B. Ichter, S. Levine, et al.π 0.5: a vision-language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    NVIDIA, L. Fan, Y . Zhu, et al. GR00T N1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    M. J. Kim, Y . Gao, T.-Y . Lin, Y .-C. Lin, Y . Ge, G. Lam, P. Liang, S. Song, M.-Y . Liu, C. Finn, and J. Gu. Cosmos policy: Fine-tuning video models for visuomotor control and planning. arXiv preprint arXiv:2601.16163, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    Ye et al

    S. Ye et al. World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026

    Pith/arXiv arXiv 2026

  6. [6]

    B. Hou, G. Li, J. Jia, T. An, X. Guo, S. Leng, H. Geng, Y . Ze, T. Harada, P. Torr, O. Mees, M. Pollefeys, Z. Liu, J. Wu, P. Abbeel, J. Malik, Y . Du, and J. Yang. World model for robot learning: A comprehensive survey.arXiv preprint arXiv:2605.00080, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    A careful examination of large behavior models for multitask dexterous manipulation.arXiv preprint arXiv:2507.05331, 2025

    TRI LBM Team. A careful examination of large behavior models for multitask dexterous manipulation.arXiv preprint arXiv:2507.05331, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Dasari, O

    S. Dasari, O. Mees, S. Zhao, M. K. Srirama, and S. Levine. The ingredients for robotic dif- fusion transformers. InProceedings of the IEEE International Conference on Robotics and Automation (ICRA), Atlanta, USA, 2025

    2025

  9. [9]

    S. Ross, G. J. Gordon, and J. A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), 2011

    2011

  10. [10]

    Kelly, C

    M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer. HG-DAgger: Interactive imitation learning with human experts. InIEEE International Conference on Robotics and Automation (ICRA), 2019

    2019

  11. [11]

    Mandlekar, D

    A. Mandlekar, D. Xu, R. Mart ´ın-Mart´ın, Y . Zhu, L. Fei-Fei, and S. Savarese. Human-in-the- loop imitation learning using remote teleoperation. InConference on Robot Learning (CoRL), 2020

    2020

  12. [12]

    X. Xu, Y . Hou, Z. Liu, and S. Song. Compliant residual DAgger: Improving real-world contact- rich manipulation with human corrections. InNeurIPS, 2025. URLhttps://arxiv.org/ abs/2506.16685

    arXiv 2025

  13. [13]

    Jiang, C

    Y . Jiang, C. Wang, R. Zhang, J. Wu, and L. Fei-Fei. TRANSIC: Sim-to-real policy transfer by learning from online correction. InCoRL, 2024. URLhttps://arxiv.org/abs/2405. 10315

    2024

  14. [14]

    Wagenmaker, M

    A. Wagenmaker, M. Nakamoto, Y . Zhang, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine. Steering your diffusion policy with latent space reinforcement learning. InConfer- ence on Robot Learning (CoRL), 2025. arXiv:2506.15799

    Pith/arXiv arXiv 2025

  15. [15]

    Ghosh, H

    Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y . Tan, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. InProceedings of Robotics: Science and Systems, Delft, Netherlands, 2024. 9

    2024

  16. [16]

    Lipman, R

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  17. [17]

    J. Pai, L. Achenbach, V . Montesinos, B. Forrai, O. Mees, and E. Nava. mimic-video: Video- action models for generalizable robot control beyond vlas. InProceedings of Robotics: Science and Systems, Sydney, Australia, 2026

    2026

  18. [18]

    H. Liu, S. Nasiriany, L. Zhang, Z. Bao, and Y . Zhu. Robot learning on the job: Human-in- the-loop autonomy and learning during deployment. InProceedings of Robotics: Science and Systems (RSS), 2023

    2023

  19. [19]

    H. Liu, Y . Zhang, V . Betala, E. Zhang, J. Liu, C. Ding, and Y . Zhu. Multi-task interactive robot fleet learning with visual world models. InConference on Robot Learning (CoRL), 2024

    2024

  20. [20]

    J. Luo, C. Xu, J. Wu, and S. Levine. Precise and dexterous robotic manipulation via human- in-the-loop reinforcement learning.arXiv preprint arXiv:2410.21845, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    J. Luo, P. Dong, Y . Zhai, Y . Ma, and S. Levine. RLIF: Interactive imitation learning as rein- forcement learning. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  22. [22]

    Silver, K

    T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling. Residual policy learning.arXiv preprint arXiv:1812.06298, 2018

    Pith/arXiv arXiv 2018

  23. [23]

    Johannink, S

    T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine. Residual reinforcement learning for robot control. InICRA, 2019. URLhttps: //arxiv.org/abs/1812.03201

    Pith/arXiv arXiv 2019

  24. [24]

    Lynch, M

    C. Lynch, M. Khansari, T. Xiao, V . Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. InConference on Robot Learning (CoRL), 2019

    2019

  25. [25]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. InInternational Con- ference on Learning Representations (ICLR), 2021

    2021

  26. [26]

    Mokady, A

    R. Mokady, A. Hertz, K. Aberman, Y . Pritch, and D. Cohen-Or. Null-text inversion for editing real images using guided diffusion models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  27. [27]

    L. Rout, Y . Chen, N. Ruiz, C. Caramanis, S. Shakkottai, and W.-S. Chu. Semantic im- age inversion and editing using rectified stochastic differential equations.arXiv preprint arXiv:2410.10792, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. InConference on Robot Learning (CoRL), 2019

    2019

  29. [29]

    Fortier, T

    D. Fortier, T. Adamson, T. Hellebrekers, T. LaScala, K. Ennin, M. Murray, A. Kolobov, and G. Mullins. Benchmarking affordance generalization with BusyBox.arXiv preprint arXiv:2602.05441, 2026

    arXiv 2026

  30. [30]

    Mandlekar, D

    A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ın-Mart´ın. What matters in learning from offline human demonstrations for robot manipulation. InConference on Robot Learning (CoRL), 2021. 10 A World-Action Model Inversion Details This appendix expands the world-action model inverter sketched ...

    2021