Robotics · arXiv:2609.18207 v1 · submitted 16 Sep 2026 · 69 references
Reinforcement Learning for Real-Time Vision-Language-Action Policies
The paper says
“Reinforcement learning fine-tuning on top of large, pretrained Vision-Language-Action (VLA) models offers promise for highly reliable robot deployment.”
more of the abstract
However, because of their scale, modern VLA models suffer from high inference latency, so the observation used to select an action is often stale by execution time, creating a distribution shift that can substantially degrade reliability and performance. Prior work has explored asynchronous policy execution to reduce the effect of latency, but these methods are mostly built on imitation learning and offer no mechanism for moving beyond the training distribution toward higher reliability. We close this gap by enabling RL fine-tuning that meets the real-time control requirements of dynamic real-world manipulation. Our approach builds on EXPO-FT, a framework for sample-efficient, reliable VLA fine-tuning with reinforcement learning, and decouples slow, expressive action generation from fast, reactive action edits: a large pretrained VLA proposes action chunks using its strong behavior prior, while a lightweight edit policy performs fast, reactive decision-making by editing actions in response to changes in state, conditioned on the latest observation. We instantiate this as Real-Time EXPO-FT, an RL framework for finetuning real-time VLA policies. On the Kinetix benchmark, Real-Time EXPO-FT enables a delayed policy to achieve the best performance among delayed and non-delayed methods in 10 out of 10 environments. On four dynamic real-world tasks, robot object passing, ball balancing, table soccer kicking, and dynamic object picking, with online robot data capped at 10 minutes, Real-Time EXPO-FT improves average policy performance from 42% to 97%, all without human intervention, demonstrating rapid, sample-efficient adaptation to challenging real-world dynamics. Website: https://pd-perry.github.io/real-time-expo-ft
from the abstract
Pith says
Machine review · deepseek-v4-flash · 17 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Real-Time EXPO-FT is a genuine methodological step for RL fine-tuning of VLAs under latency, with strong simulation evidence and promising real-world results, though the latter lack replication. This line is wrong
2 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 69 references say.
The paper says, at Section V-C, Table I, Figure 5
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
All real-world results are based on a single training run per method per task. While 30 evaluation trials are reported, the variance across independent training runs (e.g., different random seeds, hardware conditions) is unknown. RL training from small online data is inherently stochastic, and a single run can be unrepresentative.
more of objection 1
The simulation section uses 4 seeds; the same standard should be applied to real-world experiments. Without an estimate of between-run variability (e.g., mean and standard deviation over at least 3 runs, or binomial confidence intervals per run), the claimed improvements from 42% to 97% are not statistically grounded. I consider this a load-bearing issue for the central real-world claim.
The paper says, at Section IV-A, Section VII-D
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The reward signal for all real-world tasks is provided by task-specific rule-based success detectors (described in Section VII-D). The accuracy of these detectors is not empirically validated (e.g., via human-labeled hold-out data or confusion matrices). Systematic false positives or false negatives would corrupt the training signal and inflate measured success rates.
more of objection 2
While the paper acknowledges this limitation in the Discussion, given that the entire real-world training relies on these detectors, a quantitative assessment of detector robustness is necessary to support the claim that RL fine-tuning 'improves average policy performance from 42% to 97%'. A sensitivity analysis or human-verified agreement rate should be reported.
6 minor objections
The paper says, at Figure 2 (left panel)
Pith says: minor objection 1 of 6
The notation 'ad:d+C' in the figure is ambiguous; it likely means 'a_{t+d:t+d+C}'. Consider adding a subscript clarification or a table of notations in the caption.
The paper says, at Table I caption
Pith says: minor objection 2 of 6
The caption states 'Training is capped at 10 minutes of online data', but the footnotes '(10 min reached)' appear inconsistently for some cells. Clarify whether the table reports the final evaluation after 10 minutes or at the point when any method first achieves 30/30. The description in Section V-C suggests the latter, but the table layout suggests the former. Please make this explicit.
show 4 more
The paper says, at Appendix A, Figure 7
Pith says: minor objection 3 of 6
The x-axis label is 'Training Compute' but no unit is given. Since the comparison is about compute efficiency, specify what unit is being used (e.g., wall-clock time, number of policy updates, or VLA denoising calls).
The paper says, at Section V-D, Figure 6
Pith says: minor objection 4 of 6
The delay study on H17 Unicycle compares 'Ours' vs 'RTC' only. It would be informative to also show a baseline without RTC (e.g., naive replanning with delay) to isolate the benefit of the real-time edit mechanism beyond simply using RTC.
The paper says, at Table II
Pith says: minor objection 5 of 6
In the column for 'DSRL w/ RTC' under Delay=4, the Catapult entry is 49%, but the next row (Catcher v3) is 71%. Check that the formatting aligns with the correct task order (the table header lists tasks but the ordering in the first column is not alphabetical; consider matching the order used in Figure 3 for consistency).
The paper says, at Section III
Pith says: minor objection 6 of 6
The notation for the delayed observation 's_{t-d}' is introduced but the MDP formulation does not explicitly model delay. The paper acknowledges this in Section IV-A, but it would be clearer to state upfront that the policy receives observations from d steps ago, making the process non-Markovian in s_t. A brief remark in the Background would help.
What would settle it
Repeat the Dynamic Picking or Ball Balancing real-world experiment with at least 3 independent training runs for Real-Time EXPO-FT and the best baseline (EXPO-FT w/ RTC). Report mean and standard deviation of final success rates. If the means overlap within one standard error, the claim of superiority is not supported.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 17 Sep 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The paper delivers what it promises: a way to RL-finetune large VLA models despite their inference latency. The core idea—decouple slow action generation from fast, observation-conditioned editing, with the edit policy trained via RL—is clean and well-motivated. The simulation experiments are the strongest part: 10 tasks, 4 seeds, and clear improvement over all delayed baselines, including the no-delay RLPD on average. The real-world results on four dynamic tasks are impressive on the surface: 42% to 97%. The method description is thorough, with detailed appendices on architecture, hyperparameters, and baselines. The authors are transparent about limitations (human resets, task-specific detectors).
Now the soft spots, in proportion. The stress-test concern is right: every real-world method is run once per task. No error bars across independent runs. The paper reports 30 evaluation trials per run, which gives binomial confidence intervals for that specific run, but not for the between-run variance. In small-data RL, that variance can be large. The headline improvement from 42% to 97% could be real, but it could also reflect a favorable random seed or hardware conditions. The paper acknowledges this implicitly by saying 'single checkpoint' for some baselines, but does not discuss the implication for the main claim. The reward classifiers are rule-based and not validated for false positives/negatives; a systematic detector failure would corrupt the training signal. That said, the simulation evidence with 4 seeds is solid and suggests the method works under controlled conditions.
Who this is for: researchers working on VLA fine-tuning, real-time control, or sample-efficient RL for robotics. It is a competent engineering paper with a novel algorithmic combination. It deserves a serious referee; the real-world replication concern is addressable in revision (e.g., add error bars, validate detectors). I would bring it to reading group for the simulation results and the architecture design. I would cite it if I worked on real-time RL for robotics, with a note about the single-run limitation.
The strongest objection
The paper's central real-world claim—improving average success from 42% to 97%—rests on a single training run per method per task (Table I). No multiple independent runs are reported, despite the simulation using 4 seeds. Without replication, the observed improvement could reflect favorable random initialization or hardware conditions rather than systematic advantage. The 30 evaluation trials per run give binomial confidence intervals, but the between-run variance, which is typically large in RL from scratch with small data, is completely unknown. This is load-bearing because the headline real-world statistic is the core evidence that the method works in practice.
The reader's weakest_assumption focused on reward classifier robustness, which is a valid concern but not the most load-bearing because evaluation is human-verified. My concern is about statistical grounding: the real-world experiments lack multiple independent runs, making the headline improvement less credible. This aligns partially with the reader's note on missing error bars but adds specificity. The verdict remains CONDITIONAL as the paper needs additional replications to be fully convincing.
The referee's report
If the reported results are robust, this work addresses a practically important problem—VLA inference latency in dynamic manipulation—and provides a principled RL solution that combines sample-efficient fine-tuning (EXPO) with real-time chunking. The simulation experiments are thorough (10 tasks, 4 seeds, comparisons against multiple strong baselines) and the improvements are large. The real-world results, if reproducible across independent runs, would represent a meaningful step toward deploying high-capacity VLA models on real robots with tight timing constraints. The framework is modular and builds on publicly available components (EXPO-FT, training-time RTC), which aids reproducibility.
Pith's reading of the paper (a machine's summary, not the authors')
This paper addresses the challenge that large vision-language-action (VLA) models, though powerful, introduce inference delays that make their actions based on stale observations, undermining reliability in dynamic real-world tasks. The authors propose Real-Time EXPO-FT, a framework that decouples action generation into two timescales: a slow, asynchronous VLA model that proposes multiple candidate action chunks, and a lightweight, synchronous edit policy that refines and selects the best action using the most recent observation, guided by a learned Q-function. This design allows reinforcement learning to fine-tune the whole system, improving performance beyond what imitation learning alone can achieve. The method is evaluated on 10 simulated Kinetix tasks and four real-world manipulation tasks, raising average success from 42% to 97% with at most 10 minutes of online robot data, demonstrating rapid, sample-efficient adaptation without human intervention.
What carries the argument. The core mechanism is a decoupled two-timescale architecture: (1) Asynchronous action chunk generation by the base VLA, which produces N candidate action chunks from an earlier observation while incurring a delay d; (2) Synchronous action editing and selection, where a lightweight edit policy transforms each candidate based on the latest observation, and a learned Q-function chooses the chunk with the highest value for execution. This preserves the strong behavioral prior of the VLA while adding reactivity through real-time correction.
Core claim. Real-Time EXPO-FT shows that by asynchronously generating multiple action chunks from a delayed VLA model and then using a lightweight edit policy conditioned on the current observation to edit and select the highest-value chunk, reinforcement learning can effectively fine-tune real-time VLA policies. This two-timescale architecture overcomes the distribution shift caused by inference latency, enabling a delayed policy to outperform even undelayed baselines on dynamic tasks.
The premise it rests on. The rule-based success detectors for each real-world task are assumed to be accurate and robust; if they produce systematic false positives or negatives, the training signal is corrupted and the measured success rates may not reflect true task completion.
If the paper is right (Pith's inferences, not the authors')
Real-Time EXPO-FT establishes a general template for making any large, slow policy reactive through a fast edit policy trained with RL, potentially applicable beyond VLA models to other latency-sensitive control domains.
The method achieves near-perfect success (>95%) on all four tested real-world tasks after only 10 minutes of online data, suggesting that sample-efficient RL fine-tuning can now be practically deployed for dynamic manipulation with minimal human oversight.
By decoupling generation from editing, the framework allows the base VLA to be updated asynchronously without blocking control, enabling fine-tuning even when inference is too slow for real-time feedback.
The empirical result that Real-Time EXPO-FT with a 4-step delay outperforms RLPD with zero delay indicates that expressive action priors plus reactive editing can compensate for observational staleness.
The noise-level filtering technique (using a lightweight critic in noise space) reduces the computational cost of evaluating many base VLA candidates during training, making large-scale candidate sets practical.
The reliance on hand-designed rule-based success detectors for each real-world task, while effective in these experiments, may limit deployment to new tasks where such detectors are hard to specify; automating reward design or using learned classifiers could broaden applicability.
The framework currently requires human-provided environment resets; automating resets (e.g., using a separate reset policy or self-resetting environments) would remove a remaining operational bottleneck and enable fully autonomous training loops.
The edit policy's bounded additive edits (range $[-\beta,\beta]$) assume the base VLA's actions are already close to optimal; in settings where the base policy is much worse, the edit might be insufficient, suggesting a need for adaptive edit scales or hierarchical correction.
Real-Time EXPO-FT's success on Kinetix and four real-world tasks suggests it may generalize to other dynamic domains such as drone flight or fast assembly, but the empirical evidence is limited to manipulation tasks; transfer to other embodiments remains untested.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Reinforcement learning fine-tuning on top of large, pretrained Vision-Language-Action (VLA) models offers promise for highly reliable robot deployment. However, because of their scale, modern VLA models suffer from high inference latency, so the observation used to select an action is often stale by execution time, creating a distribution shift that can substantially degrade reliability and performance. Prior work has explored asynchronous policy execution to reduce the effect of latency, but these methods are mostly built on imitation learning and offer no mechanism for moving beyond the training distribution toward higher reliability. We close this gap by enabling RL fine-tuning that meets the real-time control requirements of dynamic real-world manipulation. Our approach builds on EXPO-FT, a framework for sample-efficient, reliable VLA fine-tuning with reinforcement learning, and decouples slow, expressive action generation from fast, reactive action edits: a large pretrained VLA proposes action chunks using its strong behavior prior, while a lightweight edit policy performs fast, reactive decision-making by editing actions in response to changes in state, conditioned on the latest observation. We instantiate this as Real-Time EXPO-FT, an RL framework for finetuning real-time VLA policies. On the Kinetix benchmark, Real-Time EXPO-FT enables a delayed policy to achieve the best performance among delayed and non-delayed methods in 10 out of 10 environments. On four dynamic real-world tasks, robot object passing, ball balancing, table soccer kicking, and dynamic object picking, with online robot data capped at 10 minutes, Real-Time EXPO-FT improves average policy performance from 42% to 97%, all without human intervention, demonstrating rapid, sample-efficient adaptation to challenging real-world dynamics. Website: https://pd-perry.github.io/real-time-expo-ft
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.68
“On four dynamic real-world tasks, robot object passing, ball balancing, table soccer kicking, and dynamic object picking, with online robot data capped at 10 minutes, Real-Time EXPO-FT improves average policy performance from 42% to 97%, all without human intervention, demonstrating rapid, sample-efficient adaptation to challenging real-world dynamics.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.18 · states limitation 0.14 · human participants 0.08 · controlled comparison 0.84 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.95 · mentions code or data availability 0.39
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Reinforcement learning fine-tuning on top of large, pretrained Vision-Language-Action (VLA) models offers promise for highly reliable robot deployment. | background 1.00 | 0.08 | 0.07 | hedged possibility 0.90 |
| However, because of their scale, modern VLA models suffer from high inference latency, so the observation used to select an action is often stale by execution time, creating a distribution shift that can substantially degrade reliability and performance. | background 0.93 | 0.12 | 0.03 | asserted 0.98 |
| Prior work has explored asynchronous policy execution to reduce the effect of latency, but these methods are mostly built on imitation learning and offer no mechanism for moving beyond the training distribution toward higher reliability. | gap 0.77 | 0.03 | 0.03 | asserted 1.00 |
| We close this gap by enabling RL fine-tuning that meets the real-time control requirements of dynamic real-world manipulation. | contribution 0.93 | 0.77 | 0.30 | asserted 1.00 |
| Our approach builds on EXPO-FT, a framework for sample-efficient, reliable VLA fine-tuning with reinforcement learning, and decouples slow, expressive action generation from fast, reactive action edits: a large pretrained VLA proposes action chunks using its strong behavior prior, while a lightweight edit policy performs fast, reactive decision-making by editing actions in response to changes in state, conditioned on the latest observation. | method 0.95 | 0.85 | 0.14 | asserted 0.99 |
| We instantiate this as Real-Time EXPO-FT, an RL framework for finetuning real-time VLA policies. | contribution 0.71 | 0.81 | 0.07 | asserted 0.98 |
| On the Kinetix benchmark, Real-Time EXPO-FT enables a delayed policy to achieve the best performance among delayed and non-delayed methods in 10 out of 10 environments. | result 1.00 | 0.95 | 0.23 | asserted 1.00 |
| On four dynamic real-world tasks, robot object passing, ball balancing, table soccer kicking, and dynamic object picking, with online robot data capped at 10 minutes, Real-Time EXPO-FT improves average policy performance from 42% to 97%, all without human intervention, demonstrating rapid, sample-efficient adaptation to challenging real-world dynamics. | result 1.00 | 0.96 | 0.42 | asserted 1.00 |
| Website: https://pd-perry.github.io/real-time-expo-ft | future work 0.33 | 0.13 | 0.02 | unclear 0.97 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,169 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (10)
Figures are the authors', reproduced from the paper.
References (69)
- [1] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.End-to-End Training of Deep Visuomotor Policies. 2016. arXiv:1504.00702 [cs.LG] on PithPith's reviewarXiv
- [2] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates”. In:2017 IEEE International Conference on Robotics and Automation (ICRA). 2017, pp. 3389–3396. DOI:10.1109/ICRA.2017.7989385 on PitharXiv
- [3] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochas- tic Actor. 2018. arXiv:1801.01290 [cs.LG] on PithPith's reviewarXiv
- [4] Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar.Dexterous Manip- ulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost. 2018. arXiv: 1810.06045 [cs.AI] on PithPith's reviewarXiv
- [5] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine.Soft Actor-Critic Algorithms and Applications on Pith
- [6] Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manip- ulation Data. 2020. arXiv:1911.05321 [cs.RO] on PithPith's reviewarXiv
- [7] Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. “Randomized Ensembled Double Q-Learning: Learning Fast Without a Model”. In:International Conference on Learning Representations. 2021 on Pith
- [8] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. “LoRA: Low-Rank Adaptation of Large Language Models”. In:International Conference on Learning Representations. 2022 on Pith
- [9] Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. “Efficient online reinforcement learning with offline data”. In:Proceedings of the 40th International Conference on Machine Learning. ICML’23. Honolulu, Hawaii, USA: JMLR.org, 2023 on Pith
- [10] Archit Sharma, Ahmed M. Ahmed, Rehaan Ahmad, and Chelsea Finn.Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning. 2023. arXiv:2303.01488 [cs.RO] on PithPith's reviewarXiv
- [11] Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar.Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone on Pith
- [12] Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miło´s, and Marek Cygan.Bigger, Reg- ularized, Optimistic: scaling for compute and sample- efficient continuous control. 2024. arXiv: 2405 . 16158 [cs.LG] on Pith
- [13] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma.Learning a Diffusion Model Policy from Rewards via Q-Score Matching. 2024 on Pith
- [14] Lars Ankile, Zhenyu Jiang, Rocky Duan, Guanya Shi, Pieter Abbeel, and Anusha Nagabandi.Residual Off- Policy RL for Finetuning Behavior Cloning Policies on Pith
- [15] Kevin Black, Manuel Y Galliker, and Sergey Levine. “Real-Time Execution of Action Chunking Flow Poli- cies”. In:The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2025 on Pith
- [16] Kevin Black, Allen Z. Ren, Michael Equi, and Sergey Levine.Training-Time Action Conditioning for Efficient Real-Time Chunking. 2025. arXiv: 2512 . 05964 [cs.RO] on Pith
- [17] Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao.ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy. 2025. arXiv:2502.05450 [cs.RO] on PithPith's reviewarXiv
- [18] Perry Dong, Alec M. Lessing, Annie S. Chen, and Chelsea Finn.Reinforcement Learning via Implicit Imitation Guidance. 2025. arXiv: 2506 . 07505 [cs.LG] on Pith
- [19] Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Es- mail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryach... on Pith
- [20] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y . Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin L... on Pith
show 49 more
- [21] Alexander Khazatsky et al.DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. 2025. arXiv: 2403.12945 [cs.RO] on PithPith's reviewarXiv
- [22] Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang.VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning. 2025. arXiv:2505.18719 [cs.RO] on PithPith's reviewarXiv
- [23] Jianlan Luo, Zheyuan Hu, Charles Xu, You Liang Tan, Jacob Berg, Archit Sharma, Stefan Schaal, Chelsea Finn, Abhishek Gupta, and Sergey Levine.SERL: A Software Suite for Sample-Efficient Robotic Re- inforcement Learning. 2025. arXiv: 2401 . 16013 [cs.RO] on Pith
- [24] Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine.Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning. 2025. arXiv:2410.21845 [cs.RO] on PithPith's reviewarXiv
- [25] Yunchao Ma, Yizhuang Zhou, Yunhuan Yang, Tiancai Wang, and Haoqiang Fan.Running VLAs at Real-time Speed. 2025. arXiv:2510.26742 [cs.RO] on PitharXiv
- [26] Michael Matthews, Michael Beukman, Chris Lu, and Jakob Nicolaus Foerster. “Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks”. In:The Thirteenth International Conference on Learning Representations. 2025 on Pith
- [27] NVIDIA, : Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llonto... on Pith
- [28] Allen Z. Ren, Justin Lidard, Lars Lien Ankile, An- thony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Sim- chowitz. “Diffusion Policy Policy Optimization”. In: The Thirteenth International Conference on Learning Representations. 2025 on Pith
- [29] Kohei Sendai, Maxime Alvarez, Tatsuya Matsushima, Yutaka Matsuo, and Yusuke Iwasawa.Leave No Obser- vation Behind: Real-time Correction for VLA Action Chunks. 2025. arXiv:2509.23224 [cs.RO] on PitharXiv
- [30] Gemini Robotics Team et al.Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Ad- vanced Embodied Reasoning, Thinking, and Motion Transfer. 2025. arXiv:2510.03342 [cs.RO] on PithPith's reviewarXiv
- [31] Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Naga- bandi, Abhishek Gupta, and Sergey Levine. “Steering Your Diffusion Policy with Latent Space Reinforcement Learning”. In:Conference on Robot Learning(2025) on Pith
- [32] Yantai Yang, Yuhao Wang, Zichen Wen, Luo Zhongwei, Chang Zou, Zhipeng Zhang, Chuan Wen, and Linfeng Zhang. “EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models”. In:The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2025 on Pith
- [33] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Jam... on PithPith's reviewarXiv
- [34] Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Xiang Li, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, Yu Wang, and Chao Yu. πRL: Online RL Fine-tuning for Flow- based Vision-Language-Action Models. 2026. arXiv: 2510.25889 [cs.LG] on PitharXiv
- [35] Perry Dong, Kuo-Han Hung, Tian Gao, Dorsa Sadigh, and Chelsea Finn.EXPO-FT: Sample- Efficient Reinforcement Learning Finetuning for Vision- Language-Action Models. 2026. arXiv: 2605.25477 [cs.RO] on PithPith's reviewarXiv
- [36] Perry Dong, Kuo-Han Hung, Alexander Swerdlow, Dorsa Sadigh, and Chelsea Finn.TQL: Scaling Q- Functions with Transformers by Preventing Attention Collapse. 2026. arXiv:2602.01439 [cs.LG] on PitharXiv
- [37] Perry Dong, Yueru Jia, Chelsea Finn, and Dorsa Sadigh. Q-Learning With World Models. 2026. arXiv: 2608. 17163 [cs.LG] on Pith
- [38] Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. “EXPO: Stable Reinforcement Learning with Expressive Policies”. In:The Fourteenth International Conference on Learning Representations. 2026 on Pith
- [39] Perry Dong, Ron Polonsky, Dorsa Sadigh, and Chelsea Finn.Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?2026. arXiv: 2607.27203 [cs.LG] on PithPith's reviewarXiv
- [40] Perry Dong, Alexander Swerdlow, Dorsa Sadigh, and Chelsea Finn.FASTER: Value-Guided Sampling for Fast RL. 2026. arXiv:2604.19730 [cs.LG] on PithPith's reviewarXiv
- [41] Perry Dong, Chongyi Zheng, Chelsea Finn, Dorsa Sadigh, and Benjamin Eysenbach.Value Flows. 2026. arXiv:2510.07650 [cs.LG] on PithPith's reviewarXiv
- [42] Hai Jiang, Yixian Zou, Binbin Liang, Boqian Liu, Fanman Meng, and Shuaicheng Liu.FutureRTC: Real- Time Robot Execution with Anticipatory-Conditioned Action Chunking. 2026. arXiv: 2607 . 24008 [cs.RO] on Pith
- [43] Kun Lei, Huanyu Li, Dongjie Yu, Zhenyu Wei, Lingx- iao Guo, Zhennan Jiang, Ziyu Wang, Shiyu Liang, and Huazhe Xu.RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 2026. arXiv: 2510.14830 [cs.RO] on PitharXiv
- [44] Qiyang Li and Sergey Levine. “Q-Learning with Adjoint Matching”. In:The Fourteenth International Conference on Learning Representations. 2026 on Pith
- [45] Yuxiang Lu, Zhe Liu, Xianzhe Fan, Zhenya Yang, Jinghua Hou, Junyi Li, Kaixin Ding, and Hengshuang Zhao. “FASTER: Rethinking Real-Time Flow VLAs”. In:arXiv preprint arXiv:2603.19199(2026) on PithPith's reviewarXiv
- [46] Jiahui Niu, Kefan Gu, Yucheng Zhao, Shengwen Liang, Tiancai Wang, Xing Hu, Ying Wang, and Huawei Li. Realtime-VLA FLASH: Speculative Inference Frame- work for Diffusion-based VLAs. 2026. arXiv: 2605. 13778 [cs.RO] on Pith
- [47] Sungjae Park and Shubham Tulsiani. πR2: Reactive Real-time Flow Policies. 2026. arXiv: 2607.26055 [cs.RO] on PithPith's reviewarXiv
- [48] Sarvesh Patil, Mitsuhiko Nakamoto, Manan Agarwal, Shashwat Saxena, Jesse Zhang, Giri Anantharaman, Cleah Winston, Chaoyi Pan, Douglas Chen, Nai-Chieh Huang, Zeynep Temel, Oliver Kroemer, Sergey Levine, Abhishek Gupta, Hongkai Dai, Paarth Shah, and Max Simchowitz.OGPO: Sample E... on Pith
- [49] Jiaming Tang, Yufei Sun, Yilong Zhao, Shang Yang, Yujun Lin, Zhuoyang Zhang, James Hou, Yao Lu, Zhijian Liu, and Song Han.VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference. 2026. arXiv:2512.01031 [cs.RO] on PithPith's reviewarXiv
- [50] Wenli Xiao, Haotian Lin, Andy Peng, Haoru Xue, Tairan He, Zhengyi Luo, Yuqi Xie, Fengyuan Hu, Linxi Fan, Guanya Shi, and Yuke Zhu. “Self-Improving Vision-Language-Action Models with Data Generation via Residual RL”. In:The Fourteenth International Conference on Learning Repres... on Pith
- [51] Charles Xu, Jost Tobias Springenberg, Michael Equi, Ali Amin, Adnan Esmail, Sergey Levine, and Liyiming Ke.RL Token: Bootstrapping Online RL with Vision- Language-Action Models. 2026. arXiv: 2604.23073 [cs.LG] on PithPith's reviewarXiv
- [52] Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang. ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning. 2026. arXiv: 2505. 22094 [cs.RO]. APPENDIX A. Noise-level Filtering Studies To handle the high randomness of dynamic tasks, we empirically find that a... on Pith
- [56] Environment and Base Policy:For simulation, we use the vector-state (symbolic) Kinetix benchmark, where no VLA is involved and the base policy is a pretrained state- based flow-matching policy. We evaluate on 10 environ- ments: car_launch, cartpole_thrust, catapult, catcher_v3... on Pith
- [57] Learner Configuration:The critic, filter, and edit policy use the same overall architecture as in the real-world setting, including a REDQ ensemble of 10 networks with two networks subsampled for each target estimate, LayerNorm, and hidden dimensions (256,256,256) . For simula... on Pith
- [58] State-Based Baselines: a) DSRL (state):DSRL applies SAC in the flow policy’s noise space over the frozen per-level base policy, with M= 1.5, target entropy 0, no entropy term in the Bellman backup, and no demonstration data. We evaluate three settings: no inference delay; dela... on Pith
- [59] Task Setting Description:Here, we provide detailed task settings for the four real-world tasks evaluated in our experiments, including the task objectives, success detector implementation and definition, reward function, initial-state randomization, camera configuration, and d... on Pith
- [60] Success Detectors:All success detectors operate on the robot’s own observation stream and therefore require no external instrumentation. Dynamic Picking is detected proprioceptively: a successful lift is declared when the end- effector height exceeds 0.30 while the gripper is ... on Pith
- [61] Additional Critic Inputs:For two tasks, a small number of quantities already measured by the success detector are written into unused slots of the proprioceptive state vector. These quantities are available to the critic, the noise- Q filter, and the edit policy. The base VLA’... on Pith
- [62] Observation Layout:Three tasks use one exterior view and one wrist view. The critic encoder consumes these views as a six-channel tensor. Ball Balancing instead uses a three- frame stack of the exterior view at t,t−k , and t−2k , while dropping the wrist view from the policy i... on Pith
- [63] Task Full Execution Strips:We visualize the full execution trajectories of the evaluated tasks. As shown in Figure 9, each strip illustrates the temporal progression of the task from initiation to completion, providing a qualitative view of the robot’s behavior throughout the ... on Pith
- [64] Task Initial-State Randomization Space:We visualize the task initial-state randomization space to illustrate the range of initial object positions. As shown in Figure 10, the randomized space is highlighted by the orange boxes, capturing the randomization applied to both the o... on Pith
- [65] Base Policy Initialization:We instantiate Real-Time EXPO-FT withπ0.5 [20] as the base policy. The model uses a LoRA [8] configuration with a gemma_2b_lora language backbone and a gemma_300m_lora action expert. The padded action dimension is 32, the action horizon is H= 16 , an... on Pith
- [66] Model Structure and Learned Components: a) Value function:The critic is a REDQ-style [7] ensemble of 10Q-networks with LayerNorm and three hidden layers of width 256. Every Q evaluation, including both the Bellman target and rollout-time action selection, samples two networks ... on Pith
- [67] Latency Model:We study two ways of realizing inference latency at a 30 Hz control rate. In thewall-clock condition, we set d= 0 and add 100 ms of real sleep to every sample actions call for soccer kicking, ball balancing, and object passing, approximating the compute time of r... on Pith
- [68] Optimization Hyperparameters:Table III lists the hyperparameters shared across all real-world experiments, while Table IV lists the task-specific settings. All learned components other than the base VLA, including theQ ensemble, filter critic, edit policy, and temperature, use... on Pith
- [69] Evaluation Protocol:Each method is evaluated from its final checkpoint unless otherwise specified, under the same latency condition used during training. F . Baseline Implementations a) RLPD [9]:We follow the SERL [23] setup for RLPD [9], using the same π0.5-style observation ... on Pith
- [2019] arXiv:1812.05905 [cs.LG] on PithPith's reviewarXiv
- [2024] arXiv:2412.06685 [cs.LG] on PithPith's reviewarXiv
- [2025] arXiv:2509.19301 [cs.RO] on PitharXiv
Cited by 3 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Toward Recursive Self-Improvement in Physical AI cs.RO · Oct 2026
- RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment cs.AI · Oct 2026
- What Makes a VLA Reactive? A Study of Execution Strategies for Dynamic Manipulation cs.RO · Nov 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment