Pith.

Robotics · arXiv:2609.18207 v1 · submitted 16 Sep 2026 · 69 references

Reinforcement Learning for Real-Time Vision-Language-Action Policies

Perry Dong, Kuo-Han Hung, Dorsa Sadigh, Chelsea Finn

The paper says

“Reinforcement learning fine-tuning on top of large, pretrained Vision-Language-Action (VLA) models offers promise for highly reliable robot deployment.”

more of the abstract

However, because of their scale, modern VLA models suffer from high inference latency, so the observation used to select an action is often stale by execution time, creating a distribution shift that can substantially degrade reliability and performance. Prior work has explored asynchronous policy execution to reduce the effect of latency, but these methods are mostly built on imitation learning and offer no mechanism for moving beyond the training distribution toward higher reliability. We close this gap by enabling RL fine-tuning that meets the real-time control requirements of dynamic real-world manipulation. Our approach builds on EXPO-FT, a framework for sample-efficient, reliable VLA fine-tuning with reinforcement learning, and decouples slow, expressive action generation from fast, reactive action edits: a large pretrained VLA proposes action chunks using its strong behavior prior, while a lightweight edit policy performs fast, reactive decision-making by editing actions in response to changes in state, conditioned on the latest observation. We instantiate this as Real-Time EXPO-FT, an RL framework for finetuning real-time VLA policies. On the Kinetix benchmark, Real-Time EXPO-FT enables a delayed policy to achieve the best performance among delayed and non-delayed methods in 10 out of 10 environments. On four dynamic real-world tasks, robot object passing, ball balancing, table soccer kicking, and dynamic object picking, with online robot data capped at 10 minutes, Real-Time EXPO-FT improves average policy performance from 42% to 97%, all without human intervention, demonstrating rapid, sample-efficient adaptation to challenging real-world dynamics. Website: https://pd-perry.github.io/real-time-expo-ft

from the abstract

Pith says

Machine review · deepseek-v4-flash · 17 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Real-Time EXPO-FT is a genuine methodological step for RL fine-tuning of VLAs under latency, with strong simulation evidence and promising real-world results, though the latter lack replication. This line is wrong

2 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 69 references say.

The paper says, at Section V-C, Table I, Figure 5

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

All real-world results are based on a single training run per method per task. While 30 evaluation trials are reported, the variance across independent training runs (e.g., different random seeds, hardware conditions) is unknown. RL training from small online data is inherently stochastic, and a single run can be unrepresentative.

more of objection 1

The simulation section uses 4 seeds; the same standard should be applied to real-world experiments. Without an estimate of between-run variability (e.g., mean and standard deviation over at least 3 runs, or binomial confidence intervals per run), the claimed improvements from 42% to 97% are not statistically grounded. I consider this a load-bearing issue for the central real-world claim.

The paper says, at Section IV-A, Section VII-D

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The reward signal for all real-world tasks is provided by task-specific rule-based success detectors (described in Section VII-D). The accuracy of these detectors is not empirically validated (e.g., via human-labeled hold-out data or confusion matrices). Systematic false positives or false negatives would corrupt the training signal and inflate measured success rates.

more of objection 2

While the paper acknowledges this limitation in the Discussion, given that the entire real-world training relies on these detectors, a quantitative assessment of detector robustness is necessary to support the claim that RL fine-tuning 'improves average policy performance from 42% to 97%'. A sensitivity analysis or human-verified agreement rate should be reported.

6 minor objections

The paper says, at Figure 2 (left panel)

Pith says: minor objection 1 of 6

The notation 'ad:d+C' in the figure is ambiguous; it likely means 'a_{t+d:t+d+C}'. Consider adding a subscript clarification or a table of notations in the caption.

The paper says, at Table I caption

Pith says: minor objection 2 of 6

The caption states 'Training is capped at 10 minutes of online data', but the footnotes '(10 min reached)' appear inconsistently for some cells. Clarify whether the table reports the final evaluation after 10 minutes or at the point when any method first achieves 30/30. The description in Section V-C suggests the latter, but the table layout suggests the former. Please make this explicit.

show 4 more

The paper says, at Appendix A, Figure 7

Pith says: minor objection 3 of 6

The x-axis label is 'Training Compute' but no unit is given. Since the comparison is about compute efficiency, specify what unit is being used (e.g., wall-clock time, number of policy updates, or VLA denoising calls).

The paper says, at Section V-D, Figure 6

Pith says: minor objection 4 of 6

The delay study on H17 Unicycle compares 'Ours' vs 'RTC' only. It would be informative to also show a baseline without RTC (e.g., naive replanning with delay) to isolate the benefit of the real-time edit mechanism beyond simply using RTC.

The paper says, at Table II

Pith says: minor objection 5 of 6

In the column for 'DSRL w/ RTC' under Delay=4, the Catapult entry is 49%, but the next row (Catcher v3) is 71%. Check that the formatting aligns with the correct task order (the table header lists tasks but the ordering in the first column is not alphabetical; consider matching the order used in Figure 3 for consistency).

The paper says, at Section III

Pith says: minor objection 6 of 6

The notation for the delayed observation 's_{t-d}' is introduced but the MDP formulation does not explicitly model delay. The paper acknowledges this in Section IV-A, but it would be clearer to state upfront that the policy receives observations from d steps ago, making the process non-Markovian in s_t. A brief remark in the Background would help.

What would settle it

Repeat the Dynamic Picking or Ball Balancing real-world experiment with at least 3 independent training runs for Real-Time EXPO-FT and the best baseline (EXPO-FT w/ RTC). Report mean and standard deviation of final success rates. If the means overlap within one standard error, the claim of superiority is not supported.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 17 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 29 s · 21,672 tokens read, 2,840 written
  3. Editor wrote the verdict above 11 s · 22,434 tokens read, 984 written
  4. Referee major revision: 2 major objections, 6 minor 25 s · 21,930 tokens read, 2,448 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The paper delivers what it promises: a way to RL-finetune large VLA models despite their inference latency. The core idea—decouple slow action generation from fast, observation-conditioned editing, with the edit policy trained via RL—is clean and well-motivated. The simulation experiments are the strongest part: 10 tasks, 4 seeds, and clear improvement over all delayed baselines, including the no-delay RLPD on average. The real-world results on four dynamic tasks are impressive on the surface: 42% to 97%. The method description is thorough, with detailed appendices on architecture, hyperparameters, and baselines. The authors are transparent about limitations (human resets, task-specific detectors).

Now the soft spots, in proportion. The stress-test concern is right: every real-world method is run once per task. No error bars across independent runs. The paper reports 30 evaluation trials per run, which gives binomial confidence intervals for that specific run, but not for the between-run variance. In small-data RL, that variance can be large. The headline improvement from 42% to 97% could be real, but it could also reflect a favorable random seed or hardware conditions. The paper acknowledges this implicitly by saying 'single checkpoint' for some baselines, but does not discuss the implication for the main claim. The reward classifiers are rule-based and not validated for false positives/negatives; a systematic detector failure would corrupt the training signal. That said, the simulation evidence with 4 seeds is solid and suggests the method works under controlled conditions.

Who this is for: researchers working on VLA fine-tuning, real-time control, or sample-efficient RL for robotics. It is a competent engineering paper with a novel algorithmic combination. It deserves a serious referee; the real-world replication concern is addressable in revision (e.g., add error bars, validate detectors). I would bring it to reading group for the simulation results and the architecture design. I would cite it if I worked on real-time RL for robotics, with a note about the single-run limitation.

The strongest objection

The paper's central real-world claim—improving average success from 42% to 97%—rests on a single training run per method per task (Table I). No multiple independent runs are reported, despite the simulation using 4 seeds. Without replication, the observed improvement could reflect favorable random initialization or hardware conditions rather than systematic advantage. The 30 evaluation trials per run give binomial confidence intervals, but the between-run variance, which is typically large in RL from scratch with small data, is completely unknown. This is load-bearing because the headline real-world statistic is the core evidence that the method works in practice.

The reader's weakest_assumption focused on reward classifier robustness, which is a valid concern but not the most load-bearing because evaluation is human-verified. My concern is about statistical grounding: the real-world experiments lack multiple independent runs, making the headline improvement less credible. This aligns partially with the reader's note on missing error bars but adds specificity. The verdict remains CONDITIONAL as the paper needs additional replications to be fully convincing.

The referee's report

If the reported results are robust, this work addresses a practically important problem—VLA inference latency in dynamic manipulation—and provides a principled RL solution that combines sample-efficient fine-tuning (EXPO) with real-time chunking. The simulation experiments are thorough (10 tasks, 4 seeds, comparisons against multiple strong baselines) and the improvements are large. The real-world results, if reproducible across independent runs, would represent a meaningful step toward deploying high-capacity VLA models on real robots with tight timing constraints. The framework is modular and builds on publicly available components (EXPO-FT, training-time RTC), which aids reproducibility.

Pith's reading of the paper (a machine's summary, not the authors')

This paper addresses the challenge that large vision-language-action (VLA) models, though powerful, introduce inference delays that make their actions based on stale observations, undermining reliability in dynamic real-world tasks. The authors propose Real-Time EXPO-FT, a framework that decouples action generation into two timescales: a slow, asynchronous VLA model that proposes multiple candidate action chunks, and a lightweight, synchronous edit policy that refines and selects the best action using the most recent observation, guided by a learned Q-function. This design allows reinforcement learning to fine-tune the whole system, improving performance beyond what imitation learning alone can achieve. The method is evaluated on 10 simulated Kinetix tasks and four real-world manipulation tasks, raising average success from 42% to 97% with at most 10 minutes of online robot data, demonstrating rapid, sample-efficient adaptation without human intervention.

What carries the argument. The core mechanism is a decoupled two-timescale architecture: (1) Asynchronous action chunk generation by the base VLA, which produces N candidate action chunks from an earlier observation while incurring a delay d; (2) Synchronous action editing and selection, where a lightweight edit policy transforms each candidate based on the latest observation, and a learned Q-function chooses the chunk with the highest value for execution. This preserves the strong behavioral prior of the VLA while adding reactivity through real-time correction.

Core claim. Real-Time EXPO-FT shows that by asynchronously generating multiple action chunks from a delayed VLA model and then using a lightweight edit policy conditioned on the current observation to edit and select the highest-value chunk, reinforcement learning can effectively fine-tune real-time VLA policies. This two-timescale architecture overcomes the distribution shift caused by inference latency, enabling a delayed policy to outperform even undelayed baselines on dynamic tasks.

The premise it rests on. The rule-based success detectors for each real-world task are assumed to be accurate and robust; if they produce systematic false positives or negatives, the training signal is corrupted and the measured success rates may not reflect true task completion.

If the paper is right (Pith's inferences, not the authors')

Real-Time EXPO-FT establishes a general template for making any large, slow policy reactive through a fast edit policy trained with RL, potentially applicable beyond VLA models to other latency-sensitive control domains.

The method achieves near-perfect success (>95%) on all four tested real-world tasks after only 10 minutes of online data, suggesting that sample-efficient RL fine-tuning can now be practically deployed for dynamic manipulation with minimal human oversight.

By decoupling generation from editing, the framework allows the base VLA to be updated asynchronously without blocking control, enabling fine-tuning even when inference is too slow for real-time feedback.

The empirical result that Real-Time EXPO-FT with a 4-step delay outperforms RLPD with zero delay indicates that expressive action priors plus reactive editing can compensate for observational staleness.

The noise-level filtering technique (using a lightweight critic in noise space) reduces the computational cost of evaluating many base VLA candidates during training, making large-scale candidate sets practical.

The reliance on hand-designed rule-based success detectors for each real-world task, while effective in these experiments, may limit deployment to new tasks where such detectors are hard to specify; automating reward design or using learned classifiers could broaden applicability.

The framework currently requires human-provided environment resets; automating resets (e.g., using a separate reset policy or self-resetting environments) would remove a remaining operational bottleneck and enable fully autonomous training loops.

The edit policy's bounded additive edits (range $[-\beta,\beta]$) assume the base VLA's actions are already close to optimal; in settings where the base policy is much worse, the edit might be insufficient, suggesting a need for adaptive edit scales or hierarchical correction.

Real-Time EXPO-FT's success on Kinetix and four real-world tasks suggests it may generalize to other dynamic domains such as drone flight or fast assembly, but the empirical evidence is limited to manipulation tasks; transfer to other embodiments remains untested.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Reinforcement learning fine-tuning on top of large, pretrained Vision-Language-Action (VLA) models offers promise for highly reliable robot deployment. However, because of their scale, modern VLA models suffer from high inference latency, so the observation used to select an action is often stale by execution time, creating a distribution shift that can substantially degrade reliability and performance. Prior work has explored asynchronous policy execution to reduce the effect of latency, but these methods are mostly built on imitation learning and offer no mechanism for moving beyond the training distribution toward higher reliability. We close this gap by enabling RL fine-tuning that meets the real-time control requirements of dynamic real-world manipulation. Our approach builds on EXPO-FT, a framework for sample-efficient, reliable VLA fine-tuning with reinforcement learning, and decouples slow, expressive action generation from fast, reactive action edits: a large pretrained VLA proposes action chunks using its strong behavior prior, while a lightweight edit policy performs fast, reactive decision-making by editing actions in response to changes in state, conditioned on the latest observation. We instantiate this as Real-Time EXPO-FT, an RL framework for finetuning real-time VLA policies. On the Kinetix benchmark, Real-Time EXPO-FT enables a delayed policy to achieve the best performance among delayed and non-delayed methods in 10 out of 10 environments. On four dynamic real-world tasks, robot object passing, ball balancing, table soccer kicking, and dynamic object picking, with online robot data capped at 10 minutes, Real-Time EXPO-FT improves average policy performance from 42% to 97%, all without human intervention, demonstrating rapid, sample-efficient adaptation to challenging real-world dynamics. Website: https://pd-perry.github.io/real-time-expo-ft

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.68

“On four dynamic real-world tasks, robot object passing, ball balancing, table soccer kicking, and dynamic object picking, with online robot data capped at 10 minutes, Real-Time EXPO-FT improves average policy performance from 42% to 97%, all without human intervention, demonstrating rapid, sample-efficient adaptation to challenging real-world dynamics.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.18 · states limitation 0.14 · human participants 0.08 · controlled comparison 0.84 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.95 · mentions code or data availability 0.39

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Reinforcement learning fine-tuning on top of large, pretrained Vision-Language-Action (VLA) models offers promise for highly reliable robot deployment. background 1.00 0.08 0.07 hedged possibility 0.90
However, because of their scale, modern VLA models suffer from high inference latency, so the observation used to select an action is often stale by execution time, creating a distribution shift that can substantially degrade reliability and performance. background 0.93 0.12 0.03 asserted 0.98
Prior work has explored asynchronous policy execution to reduce the effect of latency, but these methods are mostly built on imitation learning and offer no mechanism for moving beyond the training distribution toward higher reliability. gap 0.77 0.03 0.03 asserted 1.00
We close this gap by enabling RL fine-tuning that meets the real-time control requirements of dynamic real-world manipulation. contribution 0.93 0.77 0.30 asserted 1.00
Our approach builds on EXPO-FT, a framework for sample-efficient, reliable VLA fine-tuning with reinforcement learning, and decouples slow, expressive action generation from fast, reactive action edits: a large pretrained VLA proposes action chunks using its strong behavior prior, while a lightweight edit policy performs fast, reactive decision-making by editing actions in response to changes in state, conditioned on the latest observation. method 0.95 0.85 0.14 asserted 0.99
We instantiate this as Real-Time EXPO-FT, an RL framework for finetuning real-time VLA policies. contribution 0.71 0.81 0.07 asserted 0.98
On the Kinetix benchmark, Real-Time EXPO-FT enables a delayed policy to achieve the best performance among delayed and non-delayed methods in 10 out of 10 environments. result 1.00 0.95 0.23 asserted 1.00
On four dynamic real-world tasks, robot object passing, ball balancing, table soccer kicking, and dynamic object picking, with online robot data capped at 10 minutes, Real-Time EXPO-FT improves average policy performance from 42% to 97%, all without human intervention, demonstrating rapid, sample-efficient adaptation to challenging real-world dynamics. result 1.00 0.96 0.42 asserted 1.00
Website: https://pd-perry.github.io/real-time-expo-ft future work 0.33 0.13 0.02 unclear 0.97

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,169 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (10)
Figure 1 of the paper
Figure 1. Overview of Real-Time EXPO-FT. Real-Time EXPO-FT addresses the latency of large VLA models by combining slow action generation with fast reactive control. Bottom left: Task success rates before and after applying Real-Time EXPO-FT. Bottom right: Training success rates on real-world tasks compared with baselines. Abstract— Reinforcement learning fine-tuning on top of large, pretrained Vision-Language-Action (VLA) models offers promise for highly reliable robot deployment. However, because of their scale, modern VLA models suffer from high inference latency, so the observation used to…
Figure 2 of the paper
Figure 2. method Policy Inference st πVLA VLA Policy delay d (async inference) Time a1 t:t+H … a2 t:t+H aN t:t+H πedit Edit Policy … Q function Training … st⋅1 st+1 st+2 st+4 st+3 st+5 st+6 st+8 st+7 at Action Queue at+1 at+2 at+3 at at+5 at+6 at+7 at+4 Action prefix VLA Inference Asynchronously Edit and Select Best Action Synchronously Qdn ϵ1 ϵ2 ϵN … st st+1 st+2 st+3 Q πVLA ϵk Noise filtering (optional) ad:d+C Distill Q from dn Q delay d a˜ 1 t+d:t+d+C a˜ 2 t+d:t+d+C a˜ N t+d:t+d+C a˜ 1 t+d:t+d+C a˜ N t+d:t+d+C … a1 t+d:t+d+C aN t+d:t+d+C a˜ 2 t+d:t+d+C Q(st+d, a1 ) Q(st+d, aN) Q(st+d, a˜ 1…
Figure 3 of the paper
Figure 3. Training success across all 10 Kinetix [26] simulation tasks. The vertical dashed lines indicate the evaluation performance of two BC baselines, one with no inference delay and one with a 4-step delay. Each setting is run with 4 seeds. ϵ ∗ = arg max i∈[N] Q dn ψ
Figure 4 of the paper
Figure 4. Four real-world manipulation tasks in our evaluation suite: Dynamic Picking, Ball Balancing, Object Passing, and Soccer Kicking. All tasks require fast and reactive policies for dynamic environment changes. 2 4 6 8 10 Robot Data (min) 0.0 0.2 0.4 0.6 0.8 1.0 Training Success Rate Dynamic Picking 2 4 6 8 10 Robot Data (min) Ball Balancing 0.8 1.6 2.4 3.2 4 4.8 Robot Data (min) Object Passing 2 4 6 8 10 Robot Data (min) Soccer Kicking Ours EXPO-FT w/ RTC EXPO-FT w/o RTC DSRL w/ RTC DSRL w/o RTC RLPD
Figure 5 of the paper
Figure 5. Training success across four real-world dynamic tasks. We train each policy with at most 10 minutes of online robot data, or until one of the policies achieves 30/30 success during evaluation. TABLE I: Success rates on four real-world tasks. Each task is evaluated over 30 trials. We train each policy until one policy reaches a 30/30 success rate or 10 minutes of online data have been collected and used for training. Task Success Rate (x/30). Training is capped at 10 minutes of online data. SFT SFT w/ RTC RLPD DSRL DSRL w/ RTC EXPO-FT EXPO-FT w/ RTC Real-Time EXPO-FT Dynamic Picking…
Figure 6 of the paper
Figure 6. Success rates under varying delays and environment speeds. We evaluate Real-Time EXPO-FT and the baselines on the H17 Unicycle task with varying delays d and on the Object Passing task with varying passing speeds. additional experiments varying inference delays in the H17 Unicycle simulation task and object-passing speeds in the real-world Object Passing task. For varying delays, we train Real-Time EXPO-FT with different delays to simulate
Figure 7 of the paper
Figure 7. Comparison of noise-level filtering. We compare Real-Time EXPO-FT with and without noise-level filtering, where the latter uses only Q-level filtering during the Bellman backup. The x-axis shows training compute, allowing us to compare the efficiency of the two approaches. Grasp Easy Catapult Cartpole Thrust Hard Lunar Lander MJC Half Cheetah H17 Unicycle Chain Lander Catcher v3 Trampoline Car Launch
Figure 8 of the paper
Figure 8. 10 Kinetix simulation tasks [26] evaluated in our experiments. TABLE II: Full simulation results on the Kinetix benchmark: success rate (%). RL results average four random seeds × 100 evaluation episodes; BC deploys the pretrained policy without fine-tuning, and RTC adds real-time chunking on top of it (512 episodes each). Delay-4 methods replan every 4 steps under a 4-step inference delay; RLPD supports only zero delay. Bold marks every RL method within 0.95× the best RL result on that task (BC and RTC are excluded from the comparison). Task Delay = 0 Delay = 4 BC RLPD BC RTC DSRL…
Figure 9 of the paper
Figure 9. Full execution strips for the evaluated tasks. Each strip shows the temporal progression of a complete task execution,
Figure 10 of the paper
Figure 10. Initial-state randomization spaces for the real-world tasks. The orange boxes indicate the regions within which object

Figures are the authors', reproduced from the paper.

References (69)
  1. [1] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.End-to-End Training of Deep Visuomotor Policies. 2016. arXiv:1504.00702 [cs.LG] on PithPith's reviewarXiv
  2. [2] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates”. In:2017 IEEE International Conference on Robotics and Automation (ICRA). 2017, pp. 3389–3396. DOI:10.1109/ICRA.2017.7989385 on PitharXiv
  3. [3] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochas- tic Actor. 2018. arXiv:1801.01290 [cs.LG] on PithPith's reviewarXiv
  4. [4] Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar.Dexterous Manip- ulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost. 2018. arXiv: 1810.06045 [cs.AI] on PithPith's reviewarXiv
  5. [5] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine.Soft Actor-Critic Algorithms and Applications on Pith
  6. [6] Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manip- ulation Data. 2020. arXiv:1911.05321 [cs.RO] on PithPith's reviewarXiv
  7. [7] Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. “Randomized Ensembled Double Q-Learning: Learning Fast Without a Model”. In:International Conference on Learning Representations. 2021 on Pith
  8. [8] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. “LoRA: Low-Rank Adaptation of Large Language Models”. In:International Conference on Learning Representations. 2022 on Pith
  9. [9] Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. “Efficient online reinforcement learning with offline data”. In:Proceedings of the 40th International Conference on Machine Learning. ICML’23. Honolulu, Hawaii, USA: JMLR.org, 2023 on Pith
  10. [10] Archit Sharma, Ahmed M. Ahmed, Rehaan Ahmad, and Chelsea Finn.Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning. 2023. arXiv:2303.01488 [cs.RO] on PithPith's reviewarXiv
  11. [11] Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar.Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone on Pith
  12. [12] Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miło´s, and Marek Cygan.Bigger, Reg- ularized, Optimistic: scaling for compute and sample- efficient continuous control. 2024. arXiv: 2405 . 16158 [cs.LG] on Pith
  13. [13] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma.Learning a Diffusion Model Policy from Rewards via Q-Score Matching. 2024 on Pith
  14. [14] Lars Ankile, Zhenyu Jiang, Rocky Duan, Guanya Shi, Pieter Abbeel, and Anusha Nagabandi.Residual Off- Policy RL for Finetuning Behavior Cloning Policies on Pith
  15. [15] Kevin Black, Manuel Y Galliker, and Sergey Levine. “Real-Time Execution of Action Chunking Flow Poli- cies”. In:The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2025 on Pith
  16. [16] Kevin Black, Allen Z. Ren, Michael Equi, and Sergey Levine.Training-Time Action Conditioning for Efficient Real-Time Chunking. 2025. arXiv: 2512 . 05964 [cs.RO] on Pith
  17. [17] Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao.ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. 2025. arXiv:2502.05450 [cs.RO] on PithPith's reviewarXiv
  18. [18] Perry Dong, Alec M. Lessing, Annie S. Chen, and Chelsea Finn.Reinforcement Learning via Implicit Imitation Guidance. 2025. arXiv: 2506 . 07505 [cs.LG] on Pith
  19. [19] Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Es- mail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryach... on Pith
  20. [20] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y . Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin L... on Pith
show 49 more
  1. [21] Alexander Khazatsky et al.DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. 2025. arXiv: 2403.12945 [cs.RO] on PithPith's reviewarXiv
  2. [22] Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang.VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning. 2025. arXiv:2505.18719 [cs.RO] on PithPith's reviewarXiv
  3. [23] Jianlan Luo, Zheyuan Hu, Charles Xu, You Liang Tan, Jacob Berg, Archit Sharma, Stefan Schaal, Chelsea Finn, Abhishek Gupta, and Sergey Levine.SERL: A Software Suite for Sample-Efficient Robotic Re- inforcement Learning. 2025. arXiv: 2401 . 16013 [cs.RO] on Pith
  4. [24] Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine.Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. 2025. arXiv:2410.21845 [cs.RO] on PithPith's reviewarXiv
  5. [25] Yunchao Ma, Yizhuang Zhou, Yunhuan Yang, Tiancai Wang, and Haoqiang Fan.Running VLAs at Real-time Speed. 2025. arXiv:2510.26742 [cs.RO] on PitharXiv
  6. [26] Michael Matthews, Michael Beukman, Chris Lu, and Jakob Nicolaus Foerster. “Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks”. In:The Thirteenth International Conference on Learning Representations. 2025 on Pith
  7. [27] NVIDIA, : Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llonto... on Pith
  8. [28] Allen Z. Ren, Justin Lidard, Lars Lien Ankile, An- thony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Sim- chowitz. “Diffusion Policy Policy Optimization”. In: The Thirteenth International Conference on Learning Representations. 2025 on Pith
  9. [29] Kohei Sendai, Maxime Alvarez, Tatsuya Matsushima, Yutaka Matsuo, and Yusuke Iwasawa.Leave No Obser- vation Behind: Real-time Correction for VLA Action Chunks. 2025. arXiv:2509.23224 [cs.RO] on PitharXiv
  10. [30] Gemini Robotics Team et al.Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Ad- vanced Embodied Reasoning, Thinking, and Motion Transfer. 2025. arXiv:2510.03342 [cs.RO] on PithPith's reviewarXiv
  11. [31] Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Naga- bandi, Abhishek Gupta, and Sergey Levine. “Steering Your Diffusion Policy with Latent Space Reinforcement Learning”. In:Conference on Robot Learning(2025) on Pith
  12. [32] Yantai Yang, Yuhao Wang, Zichen Wen, Luo Zhongwei, Chang Zou, Zhipeng Zhang, Chuan Wen, and Linfeng Zhang. “EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models”. In:The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2025 on Pith
  13. [33] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Jam... on PithPith's reviewarXiv
  14. [34] Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Xiang Li, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, Yu Wang, and Chao Yu. πRL: Online RL Fine-tuning for Flow- based Vision-Language-Action Models. 2026. arXiv: 2510.25889 [cs.LG] on PitharXiv
  15. [35] Perry Dong, Kuo-Han Hung, Tian Gao, Dorsa Sadigh, and Chelsea Finn.EXPO-FT: Sample- Efficient Reinforcement Learning Finetuning for Vision- Language-Action Models. 2026. arXiv: 2605.25477 [cs.RO] on PithPith's reviewarXiv
  16. [36] Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, and Chelsea Finn.TQL: Scaling Q- Functions with Transformers by Preventing Attention Collapse. 2026. arXiv:2602.01439 [cs.LG] on PitharXiv
  17. [37] Perry Dong, Yueru Jia, Chelsea Finn, and Dorsa Sadigh. Q-Learning With World Models. 2026. arXiv: 2608. 17163 [cs.LG] on Pith
  18. [38] Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. “EXPO: Stable Reinforcement Learning with Expressive Policies”. In:The Fourteenth International Conference on Learning Representations. 2026 on Pith
  19. [39] Perry Dong, Ron Polonsky, Dorsa Sadigh, and Chelsea Finn.Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?2026. arXiv: 2607.27203 [cs.LG] on PithPith's reviewarXiv
  20. [40] Perry Dong, Alexander Swerdlow, Dorsa Sadigh, and Chelsea Finn.FASTER: Value-Guided Sampling for Fast RL. 2026. arXiv:2604.19730 [cs.LG] on PithPith's reviewarXiv
  21. [41] Perry Dong, Chongyi Zheng, Chelsea Finn, Dorsa Sadigh, and Benjamin Eysenbach.Value Flows. 2026. arXiv:2510.07650 [cs.LG] on PithPith's reviewarXiv
  22. [42] Hai Jiang, Yixian Zou, Binbin Liang, Boqian Liu, Fanman Meng, and Shuaicheng Liu.FutureRTC: Real- Time Robot Execution with Anticipatory-Conditioned Action Chunking. 2026. arXiv: 2607 . 24008 [cs.RO] on Pith
  23. [43] Kun Lei, Huanyu Li, Dongjie Yu, Zhenyu Wei, Lingx- iao Guo, Zhennan Jiang, Ziyu Wang, Shiyu Liang, and Huazhe Xu.RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 2026. arXiv: 2510.14830 [cs.RO] on PitharXiv
  24. [44] Qiyang Li and Sergey Levine. “Q-Learning with Adjoint Matching”. In:The Fourteenth International Conference on Learning Representations. 2026 on Pith
  25. [45] Yuxiang Lu, Zhe Liu, Xianzhe Fan, Zhenya Yang, Jinghua Hou, Junyi Li, Kaixin Ding, and Hengshuang Zhao. “FASTER: Rethinking Real-Time Flow VLAs”. In:arXiv preprint arXiv:2603.19199(2026) on PithPith's reviewarXiv
  26. [46] Jiahui Niu, Kefan Gu, Yucheng Zhao, Shengwen Liang, Tiancai Wang, Xing Hu, Ying Wang, and Huawei Li. Realtime-VLA FLASH: Speculative Inference Frame- work for Diffusion-based VLAs. 2026. arXiv: 2605. 13778 [cs.RO] on Pith
  27. [47] Sungjae Park and Shubham Tulsiani. πR2: Reactive Real-time Flow Policies. 2026. arXiv: 2607.26055 [cs.RO] on PithPith's reviewarXiv
  28. [48] Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal, Shashwat Saxena, Jesse Zhang, Giri Anantharaman, Cleah Winston, Chaoyi Pan, Douglas Chen, Nai-Chieh Huang, Zeynep Temel, Oliver Kroemer, Sergey Levine, Abhishek Gupta, Hongkai Dai, Paarth Shah, and Max Simchowitz.OGPO: Sample E... on Pith
  29. [49] Jiaming Tang, Yufei Sun, Yilong Zhao, Shang Yang, Yujun Lin, Zhuoyang Zhang, James Hou, Yao Lu, Zhijian Liu, and Song Han.VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference. 2026. arXiv:2512.01031 [cs.RO] on PithPith's reviewarXiv
  30. [50] Wenli Xiao, Haotian Lin, Andy Peng, Haoru Xue, Tairan He, Zhengyi Luo, Yuqi Xie, Fengyuan Hu, Linxi Fan, Guanya Shi, and Yuke Zhu. “Self-Improving Vision-Language-Action Models with Data Generation via Residual RL”. In:The Fourteenth International Conference on Learning Repres... on Pith
  31. [51] Charles Xu, Jost Tobias Springenberg, Michael Equi, Ali Amin, Adnan Esmail, Sergey Levine, and Liyiming Ke.RL Token: Bootstrapping Online RL with Vision- Language-Action Models. 2026. arXiv: 2604.23073 [cs.LG] on PithPith's reviewarXiv
  32. [52] Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang. ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning. 2026. arXiv: 2505. 22094 [cs.RO]. APPENDIX A. Noise-level Filtering Studies To handle the high randomness of dynamic tasks, we empirically find that a... on Pith
  33. [56] Environment and Base Policy:For simulation, we use the vector-state (symbolic) Kinetix benchmark, where no VLA is involved and the base policy is a pretrained state- based flow-matching policy. We evaluate on 10 environ- ments: car_launch, cartpole_thrust, catapult, catcher_v3... on Pith
  34. [57] Learner Configuration:The critic, filter, and edit policy use the same overall architecture as in the real-world setting, including a REDQ ensemble of 10 networks with two networks subsampled for each target estimate, LayerNorm, and hidden dimensions (256,256,256) . For simula... on Pith
  35. [58] State-Based Baselines: a) DSRL (state):DSRL applies SAC in the flow policy’s noise space over the frozen per-level base policy, with M= 1.5, target entropy 0, no entropy term in the Bellman backup, and no demonstration data. We evaluate three settings: no inference delay; dela... on Pith
  36. [59] Task Setting Description:Here, we provide detailed task settings for the four real-world tasks evaluated in our experiments, including the task objectives, success detector implementation and definition, reward function, initial-state randomization, camera configuration, and d... on Pith
  37. [60] Success Detectors:All success detectors operate on the robot’s own observation stream and therefore require no external instrumentation. Dynamic Picking is detected proprioceptively: a successful lift is declared when the end- effector height exceeds 0.30 while the gripper is ... on Pith
  38. [61] Additional Critic Inputs:For two tasks, a small number of quantities already measured by the success detector are written into unused slots of the proprioceptive state vector. These quantities are available to the critic, the noise- Q filter, and the edit policy. The base VLA’... on Pith
  39. [62] Observation Layout:Three tasks use one exterior view and one wrist view. The critic encoder consumes these views as a six-channel tensor. Ball Balancing instead uses a three- frame stack of the exterior view at t,t−k , and t−2k , while dropping the wrist view from the policy i... on Pith
  40. [63] Task Full Execution Strips:We visualize the full execution trajectories of the evaluated tasks. As shown in Figure 9, each strip illustrates the temporal progression of the task from initiation to completion, providing a qualitative view of the robot’s behavior throughout the ... on Pith
  41. [64] Task Initial-State Randomization Space:We visualize the task initial-state randomization space to illustrate the range of initial object positions. As shown in Figure 10, the randomized space is highlighted by the orange boxes, capturing the randomization applied to both the o... on Pith
  42. [65] Base Policy Initialization:We instantiate Real-Time EXPO-FT withπ0.5 [20] as the base policy. The model uses a LoRA [8] configuration with a gemma_2b_lora language backbone and a gemma_300m_lora action expert. The padded action dimension is 32, the action horizon is H= 16 , an... on Pith
  43. [66] Model Structure and Learned Components: a) Value function:The critic is a REDQ-style [7] ensemble of 10Q-networks with LayerNorm and three hidden layers of width 256. Every Q evaluation, including both the Bellman target and rollout-time action selection, samples two networks ... on Pith
  44. [67] Latency Model:We study two ways of realizing inference latency at a 30 Hz control rate. In thewall-clock condition, we set d= 0 and add 100 ms of real sleep to every sample actions call for soccer kicking, ball balancing, and object passing, approximating the compute time of r... on Pith
  45. [68] Optimization Hyperparameters:Table III lists the hyperparameters shared across all real-world experiments, while Table IV lists the task-specific settings. All learned components other than the base VLA, including theQ ensemble, filter critic, edit policy, and temperature, use... on Pith
  46. [69] Evaluation Protocol:Each method is evaluated from its final checkpoint unless otherwise specified, under the same latency condition used during training. F . Baseline Implementations a) RLPD [9]:We follow the SERL [23] setup for RLPD [9], using the same π0.5-style observation ... on Pith
  47. [2019] arXiv:1812.05905 [cs.LG] on PithPith's reviewarXiv
  48. [2024] arXiv:2412.06685 [cs.LG] on PithPith's reviewarXiv
  49. [2025] arXiv:2509.19301 [cs.RO] on PitharXiv
Cited by 3 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment