Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

ARFM turns offline RL post-training of VLA flow models into an adaptive per-batch weighting scheme that raises LIBERO average success from 88.1% to 92.1%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 10:24 UTC pith:WP7WP5Z7

load-bearing objection An empirically solid adaptive weighted-flow fine-tuning paper whose theoretical selling point is undermined by a mismatch between the loss it analyzes and the loss it runs. the 4 major comments →

arxiv 2509.04063 v1 pith:WP7WP5Z7 submitted 2025-09-04 cs.RO cs.LG

Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models

Hongyin Zhang , Shiyuan Zhang , Junxi Jin , Qixin Zeng , Yifan Qiao , Hongchao Lu , Donglin Wang This is my paper
classification cs.RO cs.LG
keywords offline reinforcement learningflow matchingvision-language-action modelsrobotic manipulationadaptive weightingpost-traininggradient variance controlLIBERO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to give vision-language-action (VLA) flow models a practical offline reinforcement-learning post-training step. Its proposal, Adaptive Reinforced Flow Matching (ARFM), is a single scaling factor alpha applied inside the flow-matching loss: each sample in a batch is reweighted by exp(alpha times its standardized RL advantage). The paper derives an objective that balances keeping enough advantage signal against keeping gradient variance low, and solves it per batch with a bisection routine. On the paper's evidence, this simple adaptive weighting overtakes the base flow policy and earlier offline RL fine-tuning methods on LIBERO, and also improves robustness to action noise, few-shot learning, and continual learning.

Core claim

Correctly scaled advantage weighting is enough to make flow-based VLA policies learn from offline RL data. ARFM standardizes leave-one-out advantages R*, estimates per-sample flow-loss variance, and each batch picks alpha to minimize weighted-gradient variance minus lambda times expected advantage. Under Gaussian assumptions on R* and flow losses, the minimizer solves 4 sqrt(x) e^{2x} - 2 sqrt(x) e^x = lambda sigma_R / sigma_L^2 with alpha = sqrt(x)/sigma_R, found by bisection. The resulting weighted flow loss fine-tunes the π0 VLA flow policy to 92.1% average LIBERO success versus 88.1% unweighted, and gives the best perturbation, few-shot, and continual-learning numbers.

What carries the argument

The load-bearing object is the adaptive scaling factor alpha in the conditional energy-weighted flow matching loss. It tilts the training distribution toward high-advantage action chunks at a strength chosen each batch, and the argument works by rewriting the choice of alpha as a bias-variance trade-off: maximizing the average advantage pushed through the softmax weight against minimizing the variance of the resulting loss gradient. The accompanying machinery is the leave-one-out advantage estimator (unbiased, critic-free R*) and a bisection solve of the first-order condition for alpha, with two Gaussian assumptions turning the objective into a closed-form equation.

Load-bearing premise

The scheme's optimal-alpha formula assumes standardized RL advantages are Gaussian and roughly independent of the flow-loss gradient, which is doubtful for sparse rewards or heavy-tailed losses, so the derived alpha is not provably optimal though the adaptive procedure may still work.

What would settle it

Take a LIBERO task suite and split expert data into two clearly separated reward groups (for example, a third of trajectories with reward 1 and the rest with reward 0), then measure the empirical distribution of the standardized advantages and flow losses in a batch. If the advantage histogram is bimodal rather than bell-shaped, the Gaussian assumption behind Corollary 2 is violated. A direct behavioral test is to compare, on the same batches, the alpha from the bisection equation against a grid search over alpha: if a grid-searched alpha yields clearly better success than the closed-form one,

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Any flow-based VLA policy that already has a return estimate can be post-trained offline by adding this one adaptive weight; no critic, no environment interaction, and no extra prediction head is required.
  • Per-batch adaptation should make fine-tuning stable when data quality varies, since alpha shrinks when gradient variance spikes and grows when the advantage signal is weak.
  • Because the weighted loss is still the standard conditional flow-matching loss, the method can be combined with other post-training techniques such as data augmentation and replay-based continual learning.
  • The reported continual-learning gains imply that advantage reweighting can soften catastrophic forgetting: samples from newly learned tasks are amplified without letting their gradients dominate the shared policy.
  • The same derivation applies to any regression-style policy loss whose per-sample gradients are available, so the method is not limited to action-chunk flow models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the authors do not pursue is to apply the same bisection rule to control weighting in plain behavior cloning or in diffusion-policy post-training; if the variance term is generic, ARFM's alpha is a ready-made schedule for any energy-weighted imitation loss.
  • The Gaussian assumptions are doing real work. With sparse success/failure rewards, R* will be bimodal rather than normal, so the closed-form alpha will be approximate; the empirical gains suggest the bounded bisection heuristic is still useful, but an apples-to-apples grid-search comparison per batch would separate the theory from the heuristic.
  • Since the leave-one-out baseline depends on batch composition, small batches will make R* noisier; the paper's few-shot results are encouraging, but ablating batch size would clarify how much the method relies on good advantage estimates versus on the adaptive alpha itself.
  • If the objective were extended to online RL, the same per-batch alpha could be used as an automatic entropy/exploitation schedule, with the advantage estimator replaced by Monte Carlo returns from environment interaction; that is a natural next step the paper only names as future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ARFM, an offline RL post-training method for vision-language-action (VLA) flow models such as π0. The method reweights the conditional flow-matching loss by an exponentiated RL advantage, with the scaling factor α adjusted per batch through a bisection procedure. The adjustment is claimed to minimize a bias-variance trade-off objective J(α)=Var(ĝ)−λS(α) that balances preserving RL advantage signal against controlling gradient variance. The authors provide two corollaries giving a closed-form optimality condition for α, and they evaluate ARFM on LIBERO multi-task, action-perturbation, few-shot, continual-learning, and real-world UR5 manipulation settings, reporting consistent improvements over π0, ReinboT, and RWR (e.g., 92.1% vs. 88.1% average LIBERO success rate).

Significance. If the theoretical derivation were sound, ARFM would be a practically valuable and simple contribution: it adapts the strength of an offline RL signal in a flow-based VLA policy in a per-batch, nearly parameter-light way, and the empirical evaluation is extensive across simulation and real-robot settings. The paper also includes useful implementation details (batch size, λ, bisection iterations, reward components) and compares against flow-matching versions of offline RL baselines. However, the central theoretical claim—that α is the minimizer of a principled variance-signal trade-off—is not supported by the derivation as written. The variance is computed for a different weighting scheme than the one implemented, and the proof invokes unstated independence and moment assumptions. The adaptive rule may still work as a heuristic, but the paper's main 'principled' contribution is currently not established.

major comments (4)
  1. [Methodology, Eq. (7) and Algorithm 2] The objective J(α) is analyzed for ĝ(α)=Σ_i \hat w_i ∇_θ L_i with \hat w_i=exp(αR_i^*) (unnormalized weights), and Corollary 1/2 solve for α under this unnormalized model. However, the practical loss L_1^τ(θ) implemented in Algorithm 2 (step 9) uses the normalized softmax weights w_i(α)=exp(αg_i)/Σ_j exp(αg_j). The gradient of the implemented loss is not the ĝ analyzed in the proof. Consequently, Eq. (7) is not the variance of the implemented gradient, and the α* obtained from Eq. (8) need not minimize J(α) for the actual training loss. The claim that α is chosen by a principled bias-variance trade-off is therefore unsupported without additional analysis of the normalized case.
  2. [Appendix, proof of Corollary 1] The variance computation treats the per-sample gradient ∇L_i as a scalar with variance σ_L^2 and uses Var((1/B)Σ exp(αR_i)∇L_i) = (m_2(α)−m_1(α)^2)σ_L^2. This identity requires, at minimum, an unstated independence assumption between R_i^* and ∇L_i and an assumption about the mean of ∇L_i; if E[∇L_i]≠0, an additional μ_L-dependent term appears and the expression is not correct. Even under Assumptions 1–2, the derived Eq. (7) does not follow from the stated definitions. The independence assumption is load-bearing and should be listed as an explicit assumption and empirically checked.
  3. [Algorithm 1] There is an inconsistency in the bisection search. Line 1 defines x_low=σ_A^2 α_min and x_high=σ_A^2 α_max, while Corollary 2 uses x=α^2 σ_R^2. If x is meant to be α^2 σ_R^2, the bounds should be σ_R^2 α_min^2 and σ_R^2 α_max^2. Also, σ_A is not defined in the text (presumably it is σ_R). As written, the bisection may not be solving Eq. (8), and the returned α* = sqrt(0.5(x_low+x_high))/σ_A does not match the derivation. This is a concrete algorithmic bug that must be fixed or explicitly clarified.
  4. [Methodology, Assumptions 1–2 and experiments] The Gaussian assumptions on R^* and on CFM losses are central to the closed-form α, but no empirical support is provided. The paper calls these assumptions 'mild and reasonable' without diagnostics. In offline RL with dense rewards and success/failure indicators, advantages are often skewed or multimodal, and flow-matching losses are typically non-Gaussian, especially at early training steps. Since Eq. (8) depends directly on these distributional assumptions, the optimality of α is not trustworthy until the assumptions are validated or a robustness analysis is given. A concrete test would be to report the empirical distributions of R_i^* and L_i^{CFM} during post-training and to compare the α from Eq. (8) with a direct grid minimization of J(α) computed on the actual normalized loss.
minor comments (5)
  1. [Abstract / Introduction] The phrase 'theoretically propose' overstates the current support; the adaptive rule should be described as a heuristic or as 'motivated by' the derived objective until the derivation is corrected.
  2. [Notation] The symbol \hat w_i is used both for exp(αR_i^*) (in J(α)) and, implicitly, for the normalized weights in L_1^τ. This ambiguity is a source of the mismatch between theory and algorithm and should be fixed by using different symbols for normalized and unnormalized weights.
  3. [Table 3] The '+4.1%', '+6.5%', and '+12.2%' values in the Avg. column are relative to π0, but the notation is inconsistent with the absolute percentages shown; please clarify whether these are absolute or relative improvements.
  4. [Section 4, real-world experiments] The real-world evaluation reports only bar-chart comparisons. Please report the number of trials per task, standard errors, and the exact success counts so the practical significance of the improvement can be assessed.
  5. [Appendix, proof of Theorem 2] The proof contains a likely typo: the expression for E[exp(−E_t(x))] writes 'exp(−βE_t(x0))' in one line where 'exp(−βE(x0))' is intended. This does not affect the theorem but should be corrected.

Circularity Check

0 steps flagged

No significant circularity: alpha is computed from per-batch statistics via an explicit trade-off objective, not fitted to test outcomes; self-citations are not load-bearing.

full rationale

The paper's central adaptive mechanism solves for alpha from current-batch statistics (sigma_R, sigma_L) through the explicit objective J(alpha)=Var−lambda S (Eq. 7, Corollary 1; Algorithm 1), and then uses that alpha in the weighted flow-matching loss. This is a per-step reweighting heuristic, not a prediction of held-out success: the alpha equation does not encode the LIBERO/UR5 success rates that are later reported, and it is evaluated against external benchmarks. The EWFM theorems (Theorems 1–2) are imported from prior work by one coauthor, but they are stated with assumptions and proven in the appendix, so they do not reduce the present claim to a self-citation. ReinboT is cited as a baseline rather than as evidence for ARFM. The main substantive weakness is a formal gap between the proof and the implemented loss: Corollary 1 computes the variance for unnormalized weights exp(alpha R) and substitutes scalar CFM-loss variance for gradient second moments, while Algorithm 2 uses normalized softmax weights and (in step 9) an extra exp(·) on R; this makes the claimed 'principled optimum' not a minimizer of the implemented objective under Assumptions 1–2. That is a correctness/validity concern, not a circular-input/output reduction, so under the hard rules it is reported here only as context and does not raise the circularity score.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central method rests on five load-bearing assumptions: the cited EWFM theorem block, two Gaussian distributional assumptions, a large-batch approximation, and an unstated independence between advantage and loss. The Gaussian assumptions are the most fragile because real RL advantages are discrete mixtures of returns and flow losses are nonnegative. The tuned lambda and reward weights are additional free inputs. No new physical entities or mediators are introduced.

free parameters (3)
  • lambda = 5e-4
    Hyperparameter balancing RL signal against gradient variance in the optimization objective J(alpha). Chosen by the authors; ablation in Fig. 4 suggests low sensitivity, but it still enters the bisection equation and affects alpha.
  • alpha bounds = [0.01, 5]
    Value range of alpha used as the bisection interval, a design choice that can clip the solution.
  • reward component weights = 13 hand-designed weights (e.g., 0.1/13, 0.01/13)
    The dense reward function used to compute advantages is assembled from manually weighted components (image MSE, SSIM, ORB, joint position, smoothness, task completion). These weights are not learned and are not shown to be robust.
axioms (5)
  • domain assumption Energy-weighted flow matching theorems (Theorems 1 and 2) from prior work
    The method inherits the theoretical equivalence between marginal and conditional energy-weighted flow matching losses from Zhang, Zhang, Gu (2025). The paper does not reprove these results, it cites them.
  • ad hoc to paper Assumption 1: Standardized RL advantages R* are Gaussian N(0, sigma_R^2)
    Methodology section, stated as an assumption. The leave-one-out advantage estimates are sums of returns and are not generally Gaussian, especially with sparse success/failure rewards.
  • ad hoc to paper Assumption 2: CFM loss values are Gaussian N(mu_L, sigma_L^2)
    Methodology section, stated as an assumption. Flow-matching losses are averages of squared errors and are typically nonnegative with heavy tails, so Gaussianity is questionable.
  • standard math Assumption 3: Large batch size allows sample approximations of expectations and variances
    Methodology section. Reasonable as a standard Monte Carlo approximation, but the required batch size for accuracy is not analyzed.
  • ad hoc to paper Implicit independence between R* and the per-sample flow loss gradient
    In the proof of Corollary 1, the variance of the weighted gradient is factored as a product of the moment of exp(alpha R*) and sigma_L^2, which requires independence between the advantage and the loss gradient. This independence is not stated or justified.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 17664 in / 11052 out tokens · 98648 ms · 2026-08-05T10:24:16.689373+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models." pith.science (2026). https://pith.science/paper/WP7WP5Z7

@misc{pith2026250904063,
  author       = {Pith},
  title        = {Pith review of: Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WP7WP5Z7}},
  note         = {Machine review of arXiv:2509.04063}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vision-Language-Action (VLA) models based on flow matching have shown excellent performance in general-purpose robotic manipulation tasks. However, the action accuracy of these models on complex downstream tasks is unsatisfactory. One important reason is that these models rely solely on the post-training paradigm of imitation learning, which makes it difficult to have a deeper understanding of the distribution properties of data quality, which is exactly what Reinforcement Learning (RL) excels at. In this paper, we theoretically propose an offline RL post-training objective for VLA flow models and induce an efficient and feasible offline RL fine-tuning algorithm -- Adaptive Reinforced Flow Matching (ARFM). By introducing an adaptively adjusted scaling factor in the VLA flow model loss, we construct a principled bias-variance trade-off objective function to optimally control the impact of RL signal on flow loss. ARFM adaptively balances RL advantage preservation and flow loss gradient variance control, resulting in a more stable and efficient fine-tuning process. Extensive simulation and real-world experimental results show that ARFM exhibits excellent generalization, robustness, few-shot learning, and continuous learning performance.

Figures

Figures reproduced from arXiv: 2509.04063 by Donglin Wang, Hongchao Lu, Hongyin Zhang, Junxi Jin, Qixin Zeng, Shiyuan Zhang, Yifan Qiao.

Figure 1
Figure 1. Figure 1: The proposed ARFM method. We study the offline [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: We evaluate performance on four categories of the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) Average SR(%) of different few-shot learning [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison of real-world pick-and [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of continuous learning performance under two task compositions (noise levels=0.1). (a) Training se [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DyGRO-VLA: Cross-Task Scaling of Vision-Language-Action Models via Dynamic Grouped Residual Optimization

    cs.RO 2026-05 unverdicted novelty 5.0

    DyGRO-VLA is a two-stage optimization framework for cross-task scaling of Vision-Language-Action models via dynamic grouped residual optimization in RL.

  2. HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensing

    cs.RO 2026-03 conditional novelty 5.0

    A vision-language-action robot policy that claims tactile-aware manipulation without tactile sensors at inference via reward-weighted flow matching and action distillation.

  3. Reflection-Based Task Adaptation for Self-Improving VLA

    cs.RO 2025-10 unverdicted novelty 5.0

    Reflective Self-Adaptation combines failure-reflective reinforcement learning with success-guided imitation learning to enable faster and more reliable task adaptation for pre-trained Vision-Language-Action models.

Reference graph

Works this paper leans on

43 extracted references · 13 canonical work pages · cited by 3 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    B.; Jaakkola, T

    Ajay, A.; Du, Y.; Gupta, A.; Tenenbaum, J. B.; Jaakkola, T. S.; and Agrawal, P. 2022. Is Conditional Generative Modeling all you need for Decision Making? In The Eleventh International Conference on Learning Representations

    work page 2022

  4. [4]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716--23736

    2022

  5. [5]

    Black, K.; Brown, N.; Driess, D.; Esmail, A.; Equi, M.; Finn, C.; Fusai, N.; Groom, L.; Hausman, K.; Ichter, B.; et al. 2024. _0 : A Vision-Language-Action Flow Model for General Robot Control. arXiv preprint arXiv:2410.24164

    Pith/arXiv arXiv 2024

  6. [6]

    Black, K.; Nakamoto, M.; Atreya, P.; Walke, H.; Finn, C.; Kumar, A.; and Levine, S. 2023. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310.10639

    Pith/arXiv arXiv 2023

  7. [7]

    Brohan, A.; Brown, N.; Carbajal, J.; Chebotar, Y.; Chen, X.; Choromanski, K.; Ding, T.; Driess, D.; Dubey, A.; Finn, C.; et al. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818

    Pith/arXiv arXiv 2023

  8. [8]

    Brohan, A.; Brown, N.; Carbajal, J.; Chebotar, Y.; Dabis, J.; Finn, C.; Gopalakrishnan, K.; Hausman, K.; Herzog, A.; Hsu, J.; et al. 2022. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817

    Pith/arXiv arXiv 2022

  9. [9]

    Cadene, R.; Alibert, S.; Soare, A.; Gallouedec, Q.; Zouitine, A.; Palma, S.; Kooijmans, P.; Aractingi, M.; Shukor, M.; Aubakirova, D.; Russi, M.; Capuano, F.; Pascale, C.; Choghari, J.; Moss, J.; and Wolf, T. 2024. LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch. https://github.com/huggingface/lerobot

    work page 2024

  10. [10]

    Chen, H.; Lu, C.; Ying, C.; Su, H.; and Zhu, J. 2022. Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling. In The Eleventh International Conference on Learning Representations

    work page 2022

  11. [11]

    Chen, L.; Lu, K.; Rajeswaran, A.; Lee, K.; Grover, A.; Laskin, M.; Abbeel, P.; Srinivas, A.; and Mordatch, I. 2021. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084--15097

    work page 2021

  12. [12]

    Chen, R. T. Q.; Rubanova, Y.; Bettencourt, J.; and Duvenaud, D. 2019. Neural Ordinary Differential Equations. arXiv:1806.07366

    Pith/arXiv arXiv 2019

  13. [13]

    K.; Tomizuka, M.; and Romeres, D

    Chen, Y.; Jha, D. K.; Tomizuka, M.; and Romeres, D. 2025. FDPP: Fine-tune Diffusion Policy with Human Preference. arXiv preprint arXiv:2501.08259

    Pith/arXiv arXiv 2025

  14. [14]

    Chi, C.; Xu, Z.; Feng, S.; Cousineau, E.; Du, Y.; Burchfiel, B.; Tedrake, R.; and Song, S. 2023. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 02783649241273668

    work page 2023

  15. [15]

    S.; Lynch, C.; Chowdhery, A.; Wahid, A.; Tompson, J.; Vuong, Q.; Yu, T.; Huang, W.; et al

    Driess, D.; Xia, F.; Sajjadi, M. S.; Lynch, C.; Chowdhery, A.; Wahid, A.; Tompson, J.; Vuong, Q.; Yu, T.; Huang, W.; et al. 2023. Palm-e: An embodied multimodal language model

    work page 2023

  16. [16]

    Guo, Y.; Zhang, J.; Chen, X.; Ji, X.; Wang, Y.-J.; Hu, Y.; and Chen, J. 2025. Improving Vision-Language-Action Model with Online Reinforcement Learning. arXiv preprint arXiv:2501.16664

    Pith/arXiv arXiv 2025

  17. [17]

    Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising Diffusion Probabilistic Models. CoRR, abs/2006.11239

    Pith/arXiv arXiv 2020

  18. [18]

    Hou, Z.; Zhang, T.; Xiong, Y.; Duan, H.; Pu, H.; Tong, R.; Zhao, C.; Zhu, X.; Qiao, Y.; Dai, J.; et al. 2025. Dita: Scaling diffusion transformer for generalist vision-language-action policy. arXiv preprint arXiv:2503.19757

    Pith/arXiv arXiv 2025

  19. [19]

    Janner, M.; Du, Y.; Tenenbaum, J.; and Levine, S. 2022. Planning with Diffusion for Flexible Behavior Synthesis. In International Conference on Machine Learning, 9902--9915. PMLR

    work page 2022

  20. [20]

    J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E.; Lam, G.; Sanketi, P.; et al

    Kim, M. J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E.; Lam, G.; Sanketi, P.; et al. 2024. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246

    Pith/arXiv arXiv 2024

  21. [21]

    Kool, W.; van Hoof, H.; and Welling, M. 2019. Buy 4 reinforce samples, get a baseline for free!

    work page 2019

  22. [22]

    R.; Fu, C.; Lunawat, I.; Sieh, I.; Kirmani, S.; et al

    Li, X.; Hsu, K.; Gu, J.; Pertsch, K.; Mees, O.; Walke, H. R.; Fu, C.; Lunawat, I.; Sieh, I.; Kirmani, S.; et al. 2024. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941

    Pith/arXiv arXiv 2024

  23. [23]

    Lipman, Y.; Chen, R. T. Q.; Ben-Hamu, H.; Nickel, M.; and Le, M. 2023. Flow Matching for Generative Modeling. arXiv:2210.02747

    Pith/arXiv arXiv 2023

  24. [24]

    Liu, B.; Zhu, Y.; Gao, C.; Feng, Y.; Liu, Q.; Zhu, Y.; and Stone, P. 2023. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. arXiv preprint arXiv:2306.03310

    Pith/arXiv arXiv 2023

  25. [25]

    Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    Pith/arXiv arXiv 2017

  26. [26]

    Lu, C.; Chen, H.; Chen, J.; Su, H.; Li, C.; and Zhu, J. 2023. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, 22825--22855. PMLR

    work page 2023

  27. [27]

    Lu, G.; Guo, W.; Zhang, C.; Zhou, Y.; Jiang, H.; Gao, Z.; Tang, Y.; and Wang, Z. 2025. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505.18719

    Pith/arXiv arXiv 2025

  28. [28]

    S.; Gao, T.; Sampaio, G

    Mark, M. S.; Gao, T.; Sampaio, G. G.; Srirama, M. K.; Sharma, A.; Finn, C.; and Kumar, A. 2024. Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone. arXiv preprint arXiv:2412.06685

    Pith/arXiv arXiv 2024

  29. [29]

    Mete, A.; Xue, H.; Wilcox, A.; Chen, Y.; and Garg, A. 2024. Quest: Self-supervised skill abstractions for learning continuous control. Advances in Neural Information Processing Systems, 37: 4062--4089

    work page 2024

  30. [30]

    Nakamoto, M.; Mees, O.; Kumar, A.; and Levine, S. 2024. Steering your generalists: Improving robotic foundation models via value guidance. arXiv preprint arXiv:2410.13816

    Pith/arXiv arXiv 2024

  31. [31]

    Peters, J.; and Schaal, S. 2007. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th International Conference on Machine Learning, ICML '07, 745–750. New York, NY, USA: Association for Computing Machinery. ISBN 9781595937933

    work page 2007

  32. [32]

    E.; Wenzel, F.; and Lioutikov, R

    Reuss, M.; Ya g murlu, \"O . E.; Wenzel, F.; and Lioutikov, R. 2024. Multimodal diffusion transformer: Learning versatile behavior from multimodal goals. arXiv preprint arXiv:2407.05996

    Pith/arXiv arXiv 2024

  33. [33]

    Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    Pith/arXiv arXiv 2017

  34. [34]

    a henb \

    Tan, S.; Dou, K.; Zhao, Y.; and Kr \"a henb \"u hl, P. 2025. Interactive Post-Training for Vision-Language-Action Models. arXiv preprint arXiv:2505.17016

    Pith/arXiv arXiv 2025

  35. [35]

    M.; Ghosh, D.; Walke, H.; Pertsch, K.; Black, K.; Mees, O.; Dasari, S.; Hejna, J.; Kreiman, T.; Xu, C.; et al

    Team, O. M.; Ghosh, D.; Walke, H.; Pertsch, K.; Black, K.; Mees, O.; Dasari, S.; Hejna, J.; Kreiman, T.; Xu, C.; et al. 2024. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213

    Pith/arXiv arXiv 2024

  36. [36]

    J.; and Zhou, M

    Wang, Z.; Hunt, J. J.; and Zhou, M. 2022. Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. In The Eleventh International Conference on Learning Representations

    work page 2022

  37. [37]

    Zhai, S.; Bai, H.; Lin, Z.; Pan, J.; Tong, P.; Zhou, Y.; Suhr, A.; Xie, S.; LeCun, Y.; Ma, Y.; et al. 2024. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in neural information processing systems, 37: 110935--110971

    work page 2024

  38. [38]

    Zhang, H.; Zhuang, Z.; Zhao, H.; Ding, P.; Lu, H.; and Wang, D. 2025. ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning. arXiv preprint arXiv:2505.07395

    Pith/arXiv arXiv 2025

  39. [39]

    Zhang, S.; Zhang, W.; and Gu, Q. 2025. Energy-Weighted Flow Matching for Offline Reinforcement Learning. arXiv:2503.04975

    Pith/arXiv arXiv 2025

  40. [40]

    Zhang, Z.; Zheng, K.; Chen, Z.; Jang, J.; Li, Y.; Wang, C.; Ding, M.; Fox, D.; and Yao, H. 2024. Grape: Generalizing robot policy via preference alignment. arXiv preprint arXiv:2411.19309

    Pith/arXiv arXiv 2024

  41. [41]

    Zhao, H.; Song, W.; Wang, D.; Tong, X.; Ding, P.; Cheng, X.; and Ge, Z. 2025. MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Models. arXiv preprint arXiv:2503.08007

    Pith/arXiv arXiv 2025

  42. [42]

    Zheng, Q.; Le, M.; Shaul, N.; Lipman, Y.; Grover, A.; and Chen, R. T. 2023. Guided flows for generative modeling and decision making. arXiv preprint arXiv:2311.13443

    Pith/arXiv arXiv 2023

  43. [43]

    Zhuang, Z.; Peng, D.; Liu, J.; Zhang, Z.; and Wang, D. 2024. Reinformer: Max-return sequence modeling for offline rl. arXiv preprint arXiv:2405.08740

    Pith/arXiv arXiv 2024