{"id":"220f3c61-a1aa-428d-afb9-b2541477d065","arxiv_id":"2608.07314","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-timescale RL post-training method that updates the semantic projection layer rarely and the action expert often improves VLA policy success on long-horizon manipulation tasks.","lead":"TEMPO is a new way to adapt vision-language-action robot models with reinforcement learning: it freezes the language-vision part and trains the two remaining modules at different speeds. The method reports better long-horizon task success than existing RL fine-tuning on the CALVIN benchmark and two real robot tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic-level TD3 bootstraps a Bellman target over transitions whose dynamics and reward depend on the faster-updated action expert, so the semantic MDP is non-stationary; the paper never justifies TD3 convergence under this coupling.","rationale":"The paper's strongest empirical evidence is the CALVIN comparison in Tables I and III: TEMPO reaches SR5 81.7 versus 78.4 for FLOWER-RL, and the 5:1 frequency ratio beats 1:1 by 4.0 points. The ablation in Table II shows both modules contribute, so the two-loop design is not vacuous. The most load-bearing assumption underneath these numbers is that the semantic-level TD3 loop is actually optimizing a well-defined value function. The reader's weakest_assumption identifies exactly this: the semantic transition depends on the action expert phi, which changes on a faster timescale. I agree with that diagnosis. I would not move the verdict to REJECT because many hierarchical and modular RL methods operate with coupled update loops and still work in practice; the concern is a theory/practice gap, not a demonstrated failure. But it is not a cosmetic issue either: Eq. (5) uses a standard Bellman target for an object (the semantic MDP) that is provably non-stationary during training, and the paper neither states the assumption explicitly nor provides a stability argument. The Monte Carlo check would settle whether the semantic critic's bootstrapping is essential for the reported gain. If it is, the paper must either justify TD3 under the coupling (e.g., bounded drift or a contractive operator) or reframe the contribution as a heuristic validated only empirically. I therefore recommend keeping CONDITIONAL: accept only once the stability assumption is tested or explicitly relaxed.","tokens_in":13568,"tokens_out":6560,"duration_ms":63905,"concrete_test":"Run TEMPO on CALVIN ABC→D with the semantic-level critic trained from Monte Carlo returns over complete instruction chains (using the same replay data, with y_t^S = R_t, no bootstrapping), keeping the 5:1 update-frequency ratio and all other hyperparameters identical. If the SR5 improvement over FLOWER-RL (approximately 3.3 points in Table I) persists, the invalidity of the Bellman bootstrap under coupled non-stationarity is not the load-bearing failure. If the improvement collapses or reverses, the semantic TD3 critic is the fragile link and the decoupling claim needs re-derivation; report at least three random seeds either way.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"In Section III-C the semantic-level loop stores transitions (h_t, z_t, r_t, h_{t+1}, d_t) and forms the Bellman target y_t^S = r_t + gamma(1-d_t) min_i Qbar_i^S(h_{t+1}, pibar_s_theta(h_{t+1})) (Eq. 5). For a fixed latent action z_t, the next semantic state h_{t+1} and reward r_t are determined by the executed action chunk a_t = pi_a_phi(z_t), so both the transition distribution and the reward function of this 'semantic MDP' depend on the current action-expert parameters phi. Because phi is updated on the faster 5:1 or 10:1 timescale while theta is updated less frequently, the Bellman target bootstraps against a moving environment: Q_s is asked to predict returns under a transition law that shifts each time the action expert changes. TD3's convergence guarantee for a stationary MDP therefore does not apply, and the paper offers no bound or empirical measure of the resulting Q_s bias or drift. The central claim that decoupled update frequencies improve long-horizon success relies on the assumption that the semantic critic remains a valid estimate of the return of pi_s_theta; if the action-expert drift is large enough, the semantic critic can chase stale value estimates and the Table III 5:1 gain could be an artifact of slower theta updates hiding critic error rather than a principled stability benefit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes TEMPO, a two-timescale reinforcement learning (RL) post-training method for vision-language-action (VLA) models. The vision-language backbone is frozen, and two modules—a semantic projection layer and an action expert—are optimized with separate TD3 loops. The key design choice is to update the semantic projection layer less frequently than the action expert, with a reported action-to-semantic update-frequency ratio ρ = 5:1 or 10:1, to reduce latent-action drift while allowing the action expert to adapt rapidly to online feedback. The method is evaluated on the CALVIN ABC→D benchmark and two real-world manipulation tasks. The paper reports that TEMPO achieves higher success rates than the pretrained FLOWER policy and a single-loop RL baseline (FLOWER-RL), and presents component ablations, frequency-ratio comparisons, and a task-set-size analysis.","tokens_in":13908,"tokens_out":3859,"duration_ms":35201,"significance":"The idea of decoupling update frequencies across functionally distinct modules in VLA post-training is plausible and relevant to the growing RL-for-VLA literature. The paper provides a clear architectural decomposition, component ablations, and real-robot experiments, which are useful empirical contributions. However, the theoretical soundness of the semantic-level TD3 loop under a non-stationary transition is not established, and the empirical evidence is weakened by the lack of variance reporting and the small absolute gains in the main comparison. If the authors can address these points, the design principle could be a valuable addition to the field; as written, the central claims are not yet fully supported.","major_comments":[{"comment":"The semantic-level TD3 loop is defined over transitions (h_t, z_t, r_t, h_{t+1}, d_t), but the transition distribution depends on the action chunk a_t = π^a_φ(z_t), and φ is updated concurrently on a faster timescale. Thus the semantic-level transition and reward functions are non-stationary during training. The Bellman target in Eq. (5) bootstraps against this shifting environment without any justification that TD3 remains valid under such coupling. This is load-bearing because the claimed stability benefit of decoupled frequencies relies on the semantic critic producing reliable value estimates. The authors should either provide a formal condition (e.g., bounded φ-drift per semantic update) or empirical diagnostics (e.g., measuring semantic-critic TD error or value estimates under varying φ) to support the method's theoretical grounding.","section":"Section III-C, Eq. (5)"},{"comment":"No variance, number of seeds, or statistical significance tests are reported for the CALVIN experiments. The headline improvement over DeFI is 0.5 percentage points on SR5 (81.7 vs 81.2), which is likely within noise. The improvement over FLOWER-RL is 3.3 points, but without error bars it is unclear whether this difference is significant. Table III shows that the 1:1 dual-loop setting is only 0.1 point below FLOWER, so the benefit of the frequency decoupling is not established. Please report mean±std over at least 3–5 seeds for all runs and provide a significance test (e.g., paired bootstrap) for the key comparisons.","section":"Tables I-IV"},{"comment":"The real-world results report only shaded regions denoting one standard deviation across three seeds and give no numerical values at the evaluation checkpoints. The description that rewards 'gradually improve and stabilize' is qualitative. To support the claim that TEMPO reaches and maintains higher rewards than FLOWER-RL, the paper should report the mean±std success values or rewards at specific training steps for both tasks, and ideally per-seed counts of successful trials.","section":"Section IV-C, Fig. 3"}],"minor_comments":[{"comment":"The baseline list in Section IV-A omits RoboFlamingo, which appears in Table I; please add a brief description or citation for it.","section":"Section IV-A"},{"comment":"The actual values of N_a, N_s, d_a, and d_s are not specified, so the frequency ratio ρ in Eq. (14) is not reproducible. Please state the per-round update counts and actor delay hyperparameters used for Tables III and IV.","section":"Section III-E"},{"comment":"The notation z_t^sg in Eq. (12) is introduced only in the text; please define it explicitly in the equation or its surrounding caption.","section":"Equations (7) and (12)"},{"comment":"The abstract claims TEMPO 'consistently outperforms' both pretrained state-of-the-art VLA models and the RL post-training baseline. Given the 0.5 percentage-point gain over DeFI and the lack of statistical analysis, this wording is too strong; consider softening to 'achieves higher success rates in our experiments.'","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting, but the empirical support is thin: the main gain is small, no variance is reported on CALVIN, and the real-robot results are qualitative. The non-stationarity of the semantic-level TD3 is a conceptual gap that should be addressed. The paper may be suitable for a robotics venue if these issues are resolved, but it needs another round of experimentation and presentation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: TEMPO's central mechanism—two module-level TD3 loops with a slower semantic projection layer and a faster action expert—is new relative to the cited work, and the frequency ablation is the right experiment to run. But the paper undersells its own uncertainty: CALVIN tables have no variance, the best gain over a strong baseline is 0.5 pp, and the semantic-level critic is trained on transitions whose dynamics depend on the fast-updated action expert. That non-stationarity is a real gap, not a fatal one.\n\nWhat's good: separating the semantic and action update rates is a clean, plausible way to preserve pretrained representations while still letting the action expert adapt. The 1:1 vs 5:1 vs 10:1 comparison in Table III is exactly the evidence you need to show the frequency decoupling matters, and it does: 1:1 dual-loop actually hurts relative to FLOWER, while 5:1 and 10:1 improve. The real-world reward curves with three seeds are a nice touch, even if the analysis is qualitative.\n\nWhere it's soft: (1) No error bars or seed counts on CALVIN. With a 3.3 pp gap over the FLOWER-RL baseline, you need to know whether that's within noise. (2) The semantic-level TD3 assumes a stationary MDP, but the transition h_t -> h_{t+1} depends on the current action expert, which is updating at a 5:1 or 10:1 rate. The Bellman target in Eq. 5 bootstraps on transitions generated by older versions of the action expert. The paper never addresses this or measures critic error. That is a missing analysis, not a known failure—but it is the kind of thing that could make the reported gains an artifact of slower semantic updates slowing down learning rather than stabilizing it. (3) No code, no hyperparameters, no task-set details beyond the table. (4) The claim of 'consistently outperforming' DeFI rests on 0.5 pp, which is probably noise.\n\nNet: worth a serious referee. I'd ask the authors for variance, code, and a section on the non-stationarity—even a simple diagnostic of semantic critic TD-error over training would help. The idea is solid enough to engage with, and the paper would be stronger if it confronted its own assumptions.","headline":"A genuinely new design idea for VLA RL post-training, with plausible ablations but underwhelming statistical evidence and an unaddressed non-stationarity in the semantic critic.","tokens_in":14420,"tokens_out":3436,"would_cite":true,"duration_ms":30067,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TEMPO claims that decoupling the update frequencies of a VLA model's semantic projection layer and action expert—slow for semantics, fast for control—improves long-horizon manipulation, reaching 81.7% success on CALVIN ABC→D.","keywords":["vision-language-action models","reinforcement learning post-training","robot manipulation","TD3","semantic-action decoupling","two-timescale updates","long-horizon manipulation","CALVIN benchmark"],"falsifier":"Run TEMPO at $\\rho=1:1$ and $\\rho=5:1$ with matched total numbers of critic updates; if the equal-frequency condition converges to the same or higher success, then the frequency decoupling is not the causal mechanism behind the reported gain.","tokens_in":13373,"feed_emoji":"🤖","tokens_out":13291,"duration_ms":97526,"temperature":0.7,"pith_summary":"TEMPO is a post-training recipe for vision-language-action (VLA) robot policies that treats semantic understanding and low-level control as separate learning problems with separate learning rates. The paper claims that freezing the pretrained vision-language backbone, adapting the semantic projection layer in a slow reinforcement-learning loop, and adapting the action expert in a fast loop gives better long-horizon manipulation than updating both modules together. This matters because the standard alternatives—supervised fine-tuning or uniform RL post-training—either suffer from distribution mismatch or destabilize the latent action space that the action head learns on. On CALVIN ABC→D the method reaches 81.7% five-task success versus 78.4% for the single-loop RL baseline, and it reaches and holds higher evaluation rewards on two real-world multi-stage tasks.","feed_headline":"Two-speed RL post-training boosts long-horizon manipulation to 81.7%","feed_subtitle":"Updating the semantic layer rarely and the action expert often beats uniform RL on CALVIN and real robots.","key_machinery":"The central mechanism is two module-level TD3 loops sharing a single rollout stream but no gradients. One loop treats the multimodal semantic representation $h_t=\\mathrm{VL}(o_t,l_t)$ as its state and the latent action $z_t=\\pi_\\theta^s(h_t)$ as its action; the other treats $z_t$ as its state and the robot action chunk $a_t=\\pi_\\phi^a(z_t)$ as the executed action. The semantic projection layer is updated every $d_s$ critic updates and the action expert every $d_a$ updates, so the relative frequency is $\\rho=f_a/f_s=N_a d_s/(N_s d_a)$; the paper finds $\\rho>1$, especially $5:1$, is what converts RL updates into long-horizon gains. Stop-gradients and separate target networks keep the two gradient pathways independent, and the frozen vision-language backbone preserves the pretrained semantics that both loops consume.","core_discovery":"TEMPO's central claim is that applying RL to a VLA policy through two module-level TD3 loops, one for the semantic projection layer $\\pi_\\theta^s$ and one for the action expert $\\pi_\\phi^a$, with an update-frequency ratio $\\rho=f_a/f_s>1$, improves long-horizon manipulation over both the pretrained policy and a single-loop RL baseline. At $\\rho=5:1$ the paper reports 100.0/97.1/92.9/87.1/81.7 percent success on CALVIN ABC→D tasks 1–5 with an average chain length of 4.59, against 99.4/96.7/91.3/85.5/78.4 for FLOWER-RL and 99.4/95.8/90.7/84.9/77.8 for the pretrained FLOWER reference. The proposed mechanism is latent-action stability: infrequent semantic updates keep the distribution of $z_t=\\pi_\\theta^s(h_t)$ comparatively stable, while frequent action-expert updates let control absorb online feedback; at the equal-frequency setting $\\rho=1:1$ the gain over FLOWER disappears, which the paper reads as evidence that the frequency decoupling, not RL updates alone, carries the improvement.","pith_inferences":["A testable extension of the claimed mechanism: log latent-action drift under $\\rho=1:1$ versus $\\rho=5:1$; if the slower semantic loop reduces drift, that directly supports the stability story.","The frequency-decoupling principle may transfer to any modular policy, not just FLOWER, and to asynchronous inference: updating semantically meaningful representations more slowly than low-level controllers is a general stability strategy.","A natural extension is to make $\\rho$ adaptive during training, starting near equality and widening as the action expert improves, which could capture the reported benefits without manual grid search.","The single-task post-training setting outperforming the 34-task setting suggests the interaction budget is diluted across tasks; adaptive allocation of rollouts by task difficulty could be a further improvement."],"forward_implications":["RL post-training can improve long-horizon VLA manipulation without backpropagating through the vision-language backbone, preserving pretrained semantic knowledge and cutting compute.","The action-to-semantic frequency ratio $\\rho$ becomes a practical tuning knob: $\\rho=1:1$ forfeits the gain, $5:1$ gives the best result, and $10:1$ remains competitive but slightly worse.","Because both module-level loops consume the same environment rollouts, decoupling update frequencies requires no extra data collection—only a change in how often each network is updated.","The gains grow as the instruction chain lengthens, from +0.6 points at SR1 to +3.9 at SR5, indicating the mechanism specifically offsets error accumulation over long horizons."],"supporting_citations":[{"why":"It supplies the twin-critic, delayed-actor TD3 algorithm used by both module-level RL loops.","marker":"[14]"},{"why":"It provides the pretrained FLOWER vision-language-flow policy that TEMPO post-trains and the reference scores it must beat.","marker":"[28]"},{"why":"It defines the CALVIN ABC→D long-horizon benchmark and the SR1–SR5 evaluation protocol used in the main comparison.","marker":"[29]"},{"why":"It motivates insulating the vision-language backbone from action-side gradients, which TEMPO operationalizes by freezing the backbone.","marker":"[12]"},{"why":"It is the closest RL post-training baseline that keeps the vision-language backbone frozen, providing contrast for the two-loop design.","marker":"[13]"},{"why":"It represents online RL post-training with PPO on LoRA, establishing the post-training setting TEMPO targets.","marker":"[9]"},{"why":"It applies uniform GRPO to all policy parameters, illustrating the single-update-strategy approach TEMPO argues against.","marker":"[10]"},{"why":"It coordinates asynchronous VLA interaction, inference, and optimization and frames the update-timing question TEMPO addresses at parameter level.","marker":"[27]"}],"fun_headline_variants":["Decoupled RL rates lift VLA manipulation to 81.7% on CALVIN","Two-rate RL decouples semantic and action learning for VLA","Slow semantic, fast action: two-timescale RL tops 81.7% on CALVIN","Slow semantic updates, fast action RL: TEMPO hits 81.7%","Decoupled two-timescale RL gives VLA a long-horizon edge (81.7%)"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the semantic-level TD3 loop's model of how latent states evolve stays valid while the action expert is being updated in the other loop.","fun_headline_variants_meta":{"raw":{"variants":["Decoupled RL rates lift VLA manipulation to 81.7% on CALVIN","Two-rate RL decouples semantic and action learning for VLA","Slow semantic, fast action: two-timescale RL tops 81.7% on CALVIN","Slow semantic updates, fast action RL: TEMPO hits 81.7%","Decoupled two-timescale RL gives VLA a long-horizon edge (81.7%)"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000866,"raw_usage":{"total_tokens":3800,"prompt_tokens":1038,"completion_tokens":2762,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":2649}},"tokens_in":654,"tokens_out":2762,"duration_ms":16780,"temperature":1.0,"reasoning_tokens":2649,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T10:25:14.033626+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TEMPO at $\\rho=1:1$ and $\\rho=5:1$ with matched total numbers of critic updates; if the equal-frequency condition converges to the same or higher success, then the frequency decoupling is not the causal mechanism behind the reported gain.","supporting_citations":[{"cited_title":"Addressing function approximation error in actor-critic methods,","cited_arxiv_id":null,"evidence_quote":"It supplies the twin-critic, delayed-actor TD3 algorithm used by both module-level RL loops."},{"cited_title":"FLOWER: Democratizing generalist robot policies with efficient vision-language-flow models,","cited_arxiv_id":null,"evidence_quote":"It provides the pretrained FLOWER vision-language-flow policy that TEMPO post-trains and the reference scores it must beat."},{"cited_title":"CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks,","cited_arxiv_id":null,"evidence_quote":"It defines the CALVIN ABC→D long-horizon benchmark and the SR1–SR5 evaluation protocol used in the main comparison."},{"cited_title":"Knowledge insulating vision-language-action models: Train fast, run fast, generalize better,","cited_arxiv_id":null,"evidence_quote":"It motivates insulating the vision-language backbone from action-side gradients, which TEMPO operationalizes by freezing the backbone."},{"cited_title":"SimpleVLA-RL: Scaling VLA training via reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"It applies uniform GRPO to all policy parameters, illustrating the single-update-strategy approach TEMPO argues against."}],"review_version":1}