{"id":"8156ffa8-2057-4c25-81ab-820d37dcf5e6","arxiv_id":"2609.18207","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Real-Time EXPO-FT achieves near-perfect success on dynamic manipulation tasks by combining asynchronous VLA action proposals with lightweight, observation-conditioned edits learned via off-policy RL.","lead":"This paper introduces Real-Time EXPO-FT, a reinforcement learning framework that fine-tunes large Vision-Language-Action (VLA) models for real-time robot control despite inference delays. The method decouples slow action generation from fast, state-aware editing, enabling a delayed VLA policy to match or exceed the performance of instant-inference baselines on highly dynamic tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Real-world results rely on a single training run per method, providing no error bars and making the reported improvement potentially unrepresentative of typical performance.","rationale":"The reader's weakest_assumption focused on reward classifier robustness, which is a valid concern but not the most load-bearing because evaluation is human-verified. My concern is about statistical grounding: the real-world experiments lack multiple independent runs, making the headline improvement less credible. This aligns partially with the reader's note on missing error bars but adds specificity. The verdict remains CONDITIONAL as the paper needs additional replications to be fully convincing.","tokens_in":21672,"tokens_out":2840,"duration_ms":29448,"concrete_test":"Repeat the Dynamic Picking or Ball Balancing real-world experiment with at least 3 independent training runs for Real-Time EXPO-FT and the best baseline (EXPO-FT w/ RTC). Report mean and standard deviation of final success rates. If the means overlap within one standard error, the claim of superiority is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central real-world claim—improving average success from 42% to 97%—rests on a single training run per method per task (Table I). No multiple independent runs are reported, despite the simulation using 4 seeds. Without replication, the observed improvement could reflect favorable random initialization or hardware conditions rather than systematic advantage. The 30 evaluation trials per run give binomial confidence intervals, but the between-run variance, which is typically large in RL from scratch with small data, is completely unknown. This is load-bearing because the headline real-world statistic is the core evidence that the method works in practice.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper proposes Real-Time EXPO-FT, a reinforcement learning framework for fine-tuning vision-language-action (VLA) models under real-time control constraints. The key idea is to decouple slow asynchronous VLA action chunk generation (which suffers from inference latency) from fast, observation-conditioned action editing via a lightweight edit policy and Q-value-based selection. The method is evaluated on 10 Kinetix simulation tasks (with 4 seeds each) and 4 real-world dynamic manipulation tasks (ball balancing, dynamic picking, object passing, soccer kicking), using π0.5 as the base VLA. In simulation, Real-Time EXPO-FT achieves a 96.2% average success rate under a 4-step delay, outperforming delayed and non-delayed baselines. In the real world, with at most 10 minutes of online data, the method raises average success from 42% to 97%, substantially exceeding prior methods such as RLPD, DSRL, and EXPO-FT.","tokens_in":21930,"tokens_out":2448,"duration_ms":25139,"significance":"If the reported results are robust, this work addresses a practically important problem—VLA inference latency in dynamic manipulation—and provides a principled RL solution that combines sample-efficient fine-tuning (EXPO) with real-time chunking. The simulation experiments are thorough (10 tasks, 4 seeds, comparisons against multiple strong baselines) and the improvements are large. The real-world results, if reproducible across independent runs, would represent a meaningful step toward deploying high-capacity VLA models on real robots with tight timing constraints. The framework is modular and builds on publicly available components (EXPO-FT, training-time RTC), which aids reproducibility.","major_comments":[{"comment":"All real-world results are based on a single training run per method per task. While 30 evaluation trials are reported, the variance across independent training runs (e.g., different random seeds, hardware conditions) is unknown. RL training from small online data is inherently stochastic, and a single run can be unrepresentative. The simulation section uses 4 seeds; the same standard should be applied to real-world experiments. Without an estimate of between-run variability (e.g., mean and standard deviation over at least 3 runs, or binomial confidence intervals per run), the claimed improvements from 42% to 97% are not statistically grounded. I consider this a load-bearing issue for the central real-world claim.","section":"Section V-C, Table I, Figure 5"},{"comment":"The reward signal for all real-world tasks is provided by task-specific rule-based success detectors (described in Section VII-D). The accuracy of these detectors is not empirically validated (e.g., via human-labeled hold-out data or confusion matrices). Systematic false positives or false negatives would corrupt the training signal and inflate measured success rates. While the paper acknowledges this limitation in the Discussion, given that the entire real-world training relies on these detectors, a quantitative assessment of detector robustness is necessary to support the claim that RL fine-tuning 'improves average policy performance from 42% to 97%'. A sensitivity analysis or human-verified agreement rate should be reported.","section":"Section IV-A, Section VII-D"}],"minor_comments":[{"comment":"The notation 'ad:d+C' in the figure is ambiguous; it likely means 'a_{t+d:t+d+C}'. Consider adding a subscript clarification or a table of notations in the caption.","section":"Figure 2 (left panel)"},{"comment":"The caption states 'Training is capped at 10 minutes of online data', but the footnotes '(10 min reached)' appear inconsistently for some cells. Clarify whether the table reports the final evaluation after 10 minutes or at the point when any method first achieves 30/30. The description in Section V-C suggests the latter, but the table layout suggests the former. Please make this explicit.","section":"Table I caption"},{"comment":"The x-axis label is 'Training Compute' but no unit is given. Since the comparison is about compute efficiency, specify what unit is being used (e.g., wall-clock time, number of policy updates, or VLA denoising calls). ","section":"Appendix A, Figure 7"},{"comment":"The delay study on H17 Unicycle compares 'Ours' vs 'RTC' only. It would be informative to also show a baseline without RTC (e.g., naive replanning with delay) to isolate the benefit of the real-time edit mechanism beyond simply using RTC.","section":"Section V-D, Figure 6"},{"comment":"In the column for 'DSRL w/ RTC' under Delay=4, the Catapult entry is 49%, but the next row (Catcher v3) is 71%. Check that the formatting aligns with the correct task order (the table header lists tasks but the ordering in the first column is not alphabetical; consider matching the order used in Figure 3 for consistency).","section":"Table II"},{"comment":"The notation for the delayed observation 's_{t-d}' is introduced but the MDP formulation does not explicitly model delay. The paper acknowledges this in Section IV-A, but it would be clearer to state upfront that the policy receives observations from d steps ago, making the process non-Markovian in s_t. A brief remark in the Background would help.","section":"Section III"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper delivers what it promises: a way to RL-finetune large VLA models despite their inference latency. The core idea—decouple slow action generation from fast, observation-conditioned editing, with the edit policy trained via RL—is clean and well-motivated. The simulation experiments are the strongest part: 10 tasks, 4 seeds, and clear improvement over all delayed baselines, including the no-delay RLPD on average. The real-world results on four dynamic tasks are impressive on the surface: 42% to 97%. The method description is thorough, with detailed appendices on architecture, hyperparameters, and baselines. The authors are transparent about limitations (human resets, task-specific detectors).\n\nNow the soft spots, in proportion. The stress-test concern is right: every real-world method is run once per task. No error bars across independent runs. The paper reports 30 evaluation trials per run, which gives binomial confidence intervals for that specific run, but not for the between-run variance. In small-data RL, that variance can be large. The headline improvement from 42% to 97% could be real, but it could also reflect a favorable random seed or hardware conditions. The paper acknowledges this implicitly by saying 'single checkpoint' for some baselines, but does not discuss the implication for the main claim. The reward classifiers are rule-based and not validated for false positives/negatives; a systematic detector failure would corrupt the training signal. That said, the simulation evidence with 4 seeds is solid and suggests the method works under controlled conditions.\n\nWho this is for: researchers working on VLA fine-tuning, real-time control, or sample-efficient RL for robotics. It is a competent engineering paper with a novel algorithmic combination. It deserves a serious referee; the real-world replication concern is addressable in revision (e.g., add error bars, validate detectors). I would bring it to reading group for the simulation results and the architecture design. I would cite it if I worked on real-time RL for robotics, with a note about the single-run limitation.","headline":"Real-Time EXPO-FT is a genuine methodological step for RL fine-tuning of VLAs under latency, with strong simulation evidence and promising real-world results, though the latter lack replication.","tokens_in":22434,"tokens_out":984,"duration_ms":10752},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Decoupling slow VLA action generation from fast reactive edits enables real-time reinforcement learning fine-tuning for dynamic robot control.","keywords":["reinforcement learning","vision-language-action models","real-time control","robot manipulation","asynchronous inference","action editing","sample efficiency","policy fine-tuning"],"falsifier":"Train Real-Time EXPO-FT on a task where the rule-based success detector is known to have at least 10% false positive rate, and compare the policy's true success (verified by human) to the detector-based evaluation; if the gap exceeds 5%, the reliance on imperfect detectors is a limiting factor.","tokens_in":24860,"feed_emoji":"🤖","tokens_out":2738,"duration_ms":51541,"temperature":0.7,"pith_summary":"This paper addresses the challenge that large vision-language-action (VLA) models, though powerful, introduce inference delays that make their actions based on stale observations, undermining reliability in dynamic real-world tasks. The authors propose Real-Time EXPO-FT, a framework that decouples action generation into two timescales: a slow, asynchronous VLA model that proposes multiple candidate action chunks, and a lightweight, synchronous edit policy that refines and selects the best action using the most recent observation, guided by a learned Q-function. This design allows reinforcement learning to fine-tune the whole system, improving performance beyond what imitation learning alone can achieve. The method is evaluated on 10 simulated Kinetix tasks and four real-world manipulation tasks, raising average success from 42% to 97% with at most 10 minutes of online robot data, demonstrating rapid, sample-efficient adaptation without human intervention.","feed_headline":"RL fine-tunes slow VLA policies to 97% success in real time","feed_subtitle":"Decoupling action generation from reactive edits overcomes inference delay, boosting dynamic manipulation with only 10 minutes of online…","key_machinery":"The core mechanism is a decoupled two-timescale architecture: (1) Asynchronous action chunk generation by the base VLA, which produces N candidate action chunks from an earlier observation while incurring a delay d; (2) Synchronous action editing and selection, where a lightweight edit policy transforms each candidate based on the latest observation, and a learned Q-function chooses the chunk with the highest value for execution. This preserves the strong behavioral prior of the VLA while adding reactivity through real-time correction.","core_discovery":"Real-Time EXPO-FT shows that by asynchronously generating multiple action chunks from a delayed VLA model and then using a lightweight edit policy conditioned on the current observation to edit and select the highest-value chunk, reinforcement learning can effectively fine-tune real-time VLA policies. This two-timescale architecture overcomes the distribution shift caused by inference latency, enabling a delayed policy to outperform even undelayed baselines on dynamic tasks.","pith_inferences":["The reliance on hand-designed rule-based success detectors for each real-world task, while effective in these experiments, may limit deployment to new tasks where such detectors are hard to specify; automating reward design or using learned classifiers could broaden applicability.","The framework currently requires human-provided environment resets; automating resets (e.g., using a separate reset policy or self-resetting environments) would remove a remaining operational bottleneck and enable fully autonomous training loops.","The edit policy's bounded additive edits (range $[-\\beta,\\beta]$) assume the base VLA's actions are already close to optimal; in settings where the base policy is much worse, the edit might be insufficient, suggesting a need for adaptive edit scales or hierarchical correction.","Real-Time EXPO-FT's success on Kinetix and four real-world tasks suggests it may generalize to other dynamic domains such as drone flight or fast assembly, but the empirical evidence is limited to manipulation tasks; transfer to other embodiments remains untested."],"forward_implications":["Real-Time EXPO-FT establishes a general template for making any large, slow policy reactive through a fast edit policy trained with RL, potentially applicable beyond VLA models to other latency-sensitive control domains.","The method achieves near-perfect success (>95%) on all four tested real-world tasks after only 10 minutes of online data, suggesting that sample-efficient RL fine-tuning can now be practically deployed for dynamic manipulation with minimal human oversight.","By decoupling generation from editing, the framework allows the base VLA to be updated asynchronously without blocking control, enabling fine-tuning even when inference is too slow for real-time feedback.","The empirical result that Real-Time EXPO-FT with a 4-step delay outperforms RLPD with zero delay indicates that expressive action priors plus reactive editing can compensate for observational staleness.","The noise-level filtering technique (using a lightweight critic in noise space) reduces the computational cost of evaluating many base VLA candidates during training, making large-scale candidate sets practical."],"supporting_citations":[{"why":"Introduces real-time chunking (RTC) for asynchronous VLA execution with action inpainting, which Real-Time EXPO-FT builds upon for base policy generation.","marker":"[15]"},{"why":"Proposes training-time RTC, which conditions the VLA on the committed action prefix during inference; Real-Time EXPO-FT uses this formulation for VLA action generation.","marker":"[16]"},{"why":"EXPO-FT is the base sample-efficient RL fine-tuning framework for VLA models; Real-Time EXPO-FT extends it with delay-aware real-time execution.","marker":"[35]"},{"why":"EXPO provides the core RL algorithm (edit policy + Q-function) for fine-tuning expressive policies, which Real-Time EXPO-FT adapts for real-time control.","marker":"[38]"},{"why":"RLPD is a sample-efficient off-policy RL baseline; the paper compares against it to show that Real-Time EXPO-FT outperforms even latency-free learned policies.","marker":"[9]"},{"why":"DSRL is a recent RL method for finetuning diffusion/flow policies in latent noise space; used as a delayed RL baseline for comparison.","marker":"[31]"},{"why":"Provides the VLA base model $\\pi_0.5$ used in real-world experiments; its inference latency motivates the need for real-time RL fine-tuning.","marker":"[19]"},{"why":"Provides the $\\pi_0.5$ model and its LoRA fine-tuning setup, which is used as the base policy in Real-Time EXPO-FT.","marker":"[20]"}],"fun_headline_variants":["RL cures stale observations with a two-timescale edit","Delayed VLA policies hit 97% via RL action edits","10-minute RL fine-tuning lifts robot success 42% to 97%","Lightweight edit policy turns slow VLA into real-time champ","RL overcomes inference delay, matches undelayed baselines"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The rule-based success detectors for each real-world task are assumed to be accurate and robust; if they produce systematic false positives or negatives, the training signal is corrupted and the measured success rates may not reflect true task completion.","fun_headline_variants_meta":{"raw":{"variants":["RL cures stale observations with a two-timescale edit","Delayed VLA policies hit 97% via RL action edits","10-minute RL fine-tuning lifts robot success 42% to 97%","Lightweight edit policy turns slow VLA into real-time champ","RL overcomes inference delay, matches undelayed baselines"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"cost":0.000203448,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000203448,"upstream_inference_prompt_cost":0.000107128,"upstream_inference_completions_cost":9.632e-05},"total_tokens":1314,"prompt_tokens":970,"completion_tokens":344,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":254}},"tokens_in":970,"tokens_out":344,"cost_source":"openrouter_usage","duration_ms":4942,"image_pages":[],"temperature":1.0,"reasoning_tokens":254,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-17T10:58:51.042117+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Train Real-Time EXPO-FT on a task where the rule-based success detector is known to have at least 10% false positive rate, and compare the policy's true success (verified by human) to the detector-based evaluation; if the gap exceeds 5%, the reliance on imperfect detectors is a limiting factor.","supporting_citations":[{"cited_title":"Real-Time Execution of Action Chunking Flow Poli- cies","cited_arxiv_id":null,"evidence_quote":"Introduces real-time chunking (RTC) for asynchronous VLA execution with action inpainting, which Real-Time EXPO-FT builds upon for base policy generation."},{"cited_title":"Ren, Michael Equi, and Sergey Levine.Training-Time Action Conditioning for Efficient Real-Time Chunking","cited_arxiv_id":null,"evidence_quote":"Proposes training-time RTC, which conditions the VLA on the committed action prefix during inference; Real-Time EXPO-FT uses this formulation for VLA action generation."},{"cited_title":"EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models","cited_arxiv_id":"2605.25477","evidence_quote":"EXPO-FT is the base sample-efficient RL fine-tuning framework for VLA models; Real-Time EXPO-FT extends it with delay-aware real-time execution."},{"cited_title":"EXPO: Stable Reinforcement Learning with Expressive Policies","cited_arxiv_id":null,"evidence_quote":"EXPO provides the core RL algorithm (edit policy + Q-function) for fine-tuning expressive policies, which Real-Time EXPO-FT adapts for real-time control."},{"cited_title":"Efficient online reinforcement learning with offline data","cited_arxiv_id":null,"evidence_quote":"RLPD is a sample-efficient off-policy RL baseline; the paper compares against it to show that Real-Time EXPO-FT outperforms even latency-free learned policies."},{"cited_title":"Steering Your Diffusion Policy with Latent Space Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"DSRL is a recent RL method for finetuning diffusion/flow policies in latent noise space; used as a delayed RL baseline for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VLA base model $\\pi_0.5$ used in real-world experiments; its inference latency motivates the need for real-time RL fine-tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the $\\pi_0.5$ model and its LoRA fine-tuning setup, which is used as the base policy in Real-Time EXPO-FT."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}