{"id":"81a35744-3d94-41d4-92b5-d4a20cd844be","arxiv_id":"2611.03103","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Moving-target VLA manipulation loses about half its success at only 100 ms of inference latency; a benchmark and a two-parameter speed-scaling model quantify and predict this loss.","lead":"VLALagBench adds controlled inference delays to robot manipulation tasks and measures how often three vision-language-action policies still succeed. The paper finds that even 100 ms of latency can roughly halve success when targets move, and it builds a model that predicts success so system builders can choose execution settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Section 5.2's deployment-selection claim is validated on exactly one cell (π0.5, set-down, 0.075 m/s); the four 'systems' only vary latency and throughput over the same MuJoCo surface, so cross-task and cross-policy generality is unestablished.","rationale":"The direct empirical claim — that 100 ms of inference latency can nearly halve success on moving targets — is well supported by 200-rollout conditions with Wilson intervals, cross-simulator checks in Isaac Sim, and real-robot verification, so I do not object to Finding 1. The empirical status of Eq. (1) is not by itself the decisive problem: the paper reports held-out validation, compares RMS error to a 4.0-point sampling-noise floor, and explicitly lists the GR00T N1.7 and near-static async departures. The unresolved gap is that the model's advertised use for system-specific configuration selection is demonstrated in only one policy-task-speed cell, with the four 'systems' differing only by latency and rate limit over the same simulator surface. My proposed check would settle whether the selection protocol generalizes beyond that cell. This reinforces, rather than moves, the reader's CONDITIONAL verdict: the benchmark measurements are credible, but the model-guided selection contribution needs broader validation and the release of artifacts before the abstract-level claim is fully supported.","tokens_in":32495,"tokens_out":11054,"duration_ms":265337,"concrete_test":"Run the exact Table 3 calibration protocol — one complete continuous-100 ms curve plus one synchronous and one continuous point at 200 ms — for π0.5 on release and grasping and for MolmoAct2 on set-down, at both 0.075 and 0.1 m/s, and compare the predicted best (mode, H) against the measured best under the same four latency/throughput constraints as Systems A–D. Report the match rate over these cells; if the model selects the measured best configuration in fewer than 75% of cells, or misses by more than 5 success-rate points in more than two cells, the system-selection claim should be downgraded to a single-case demonstration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's third contribution — using the §5.1 model to select execution mode and horizon for a given inference system — is supported by a single validation cell. Table 3 calibrates the model on π0.5, set-down, target speed 0.075 m/s, using one complete continuous-100 ms curve plus two calibration points, and then evaluates Systems A–D on that same task, policy, and speed. The differences between the four systems are only the injected latency and the per-robot throughput cap; the underlying success surface is the same MuJoCo set-down surface. The experiment therefore demonstrates that Eq. (1) can rank four latency/rate operating points for one policy-task-speed combination, not that it selects configurations across tasks, policies, or speeds. Because a and b in Eq. (1) are refit for every policy-task pair and the reference curve is measured per task, the abstract's general 'guide the selection' claim needs cross-task and cross-policy evidence. The acknowledged failures for GR00T N1.7 (Table 2) and near-static async at 250 ms (Appendix D) are honestly reported and are not the basis of this concern; they make the lack of replication in the flagship deployment validation more consequential.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"VLALagBench is a benchmark and measurement study that systematically varies task type, target speed, execution mode, and injected inference latency for three VLA policies on four MuJoCo/LIBERO tasks, with secondary Isaac Sim and real-robot checks. The paper reports that 100 ms latency can nearly halve success on moving targets, that the loss arises from stale observations and robot pauses with task-dependent weight, that asynchronous execution recovers part of the loss, and that policies differ in sensitivity. It then proposes an empirical speed-scaling model (Eq. 1) that predicts success at unseen latencies from limited calibration data, and uses the model to select execution modes and horizons for four hypothetical deployment systems. The evaluation uses 200 rollouts per condition, Wilson intervals, and held-out latency predictions with an estimated sampling-noise floor.","tokens_in":32877,"tokens_out":6392,"duration_ms":50121,"significance":"If the findings hold, the benchmark fills a real gap: existing dynamic-manipulation benchmarks do not systematically isolate inference latency while varying task type, speed, execution mode, and policy. The strengths are substantial: controlled latency injection, 200 rollouts per condition, 95% Wilson intervals, held-out latency predictions whose RMS error is reported against an estimated sampling-noise floor, and honest, clearly located discussion of the model's failures for GR00T N1.7 and near-static asynchronous behavior at 250 ms. The paper also verifies the main findings in a second simulator and on a real robot. The main weakness is that the third contribution, system-specific configuration selection, is validated on exactly one policy-task-speed cell, while the abstract states the guidance claim in general terms. The empirical model itself is an ansatz rather than a derivation, but it is evaluated on held-out latencies; the concern is scope, not circularity of the core measurements.","major_comments":[{"comment":"The deployment-selection contribution is validated on exactly one policy-task-speed cell. In §5.2, the model is calibrated for π0.5 on set-down at 0.075 m/s from one complete continuous-100 ms curve plus two 200 ms points, and Systems A–D differ only in injected latency and per-robot rate limit over the same MuJoCo success surface. System A's selected configuration is itself the calibration curve, so its row in Table 3 is not an independent validation point. This demonstrates that Eq. (1) can rank four latency/throughput operating points for one cell, but not the abstract's general claim that the predictions 'guide the selection of system-specific VLA algorithm configurations.' Because a and b are refitted for every policy-task pair and the reference curve is measured per task, cross-task and cross-policy generalization does not follow automatically. Please add at least one additional policy-task cell to the §5.2 validation, ideally under a rate-limited configuration, or explicitly restrict the deployment-selection claim to the demonstrated cell.","section":"§5.2 / Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The headline accuracy numbers (5.7–7.8 points RMS) are reported for π0.5 and MolmoAct2, while GR00T N1.7's limited-data calibration has a median RMS error of 11.6 points and fitted coefficients whose signs contradict the proposed staleness and idle-time mechanisms on set-down and grasping (Table 2). Appendix D also documents two systematic departures that Eq. (1) cannot represent: at 250 ms, lazy and continuous execution lose near-static success on release and set-down, and on the fastest targets synchronous execution beats asynchronous execution despite the model's prediction that synchronous execution should be worse. These limitations are honestly acknowledged, but because Eq. (1) is the foundation of the §5.2 selection claim, the model's validity domain should be stated as a scope condition in the abstract and Section 5 rather than only in appendix caveats. A concrete test would be a deployment-selection validation on a policy-task pair where Eq. (1) actually fits, together with a statement of which of the paper's conclusions depend on the model rather than on the direct measurements.","section":"§5.1 / Eq. (1), Table 2, Appendix D","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The benchmark name is spelled both 'VLALagBench' and 'VLALAGBENCH'; please standardize the spelling in the abstract, main text, and figures.","section":"Throughout","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The statement that synchronous and lazy asynchronous execution have identical average observation staleness is central to the pause-vs-staleness attribution; a one-line algebraic derivation in the text would make this easier to verify than reading it off Figure 5.","section":"§3.5 / Figure 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The latencies for Systems B–D combine measured GPU latencies with network latency estimates taken from VLA-Perf rather than measured in the deployment scenario; a short sensitivity analysis over the network estimate would strengthen the recommendation.","section":"Table 3 / Appendix D","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The real-robot results use five rollouts per condition, which is appropriate for a pilot but should be explicitly labeled as a pilot verification in the main text rather than appearing only in the appendix.","section":"Appendix F","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Two things to know up front. The measurements are well built: 200 rollouts per condition, Wilson intervals, latency injected as control steps with the action budget extended so the policy isn't starved, and a stated sampling-noise floor that the model is judged against. The headline — 100 ms of latency can nearly halve success on moving targets, with sensitivity varying by task and policy — is direct measurement, not a fitted artifact.\n\nWhat's new: ReflexBench already varied latency but on one policy and two tasks; MoveBench and DOM don't sweep latency at all. This paper is the first joint controlled sweep of task type, target speed, execution mode, and three generalist policies, and it turns the vague intuition about latency into two measurable mechanisms: stale observations hurt precision-sensitive tasks, robot pauses hurt continuous-motion tasks. The tolerance-zone analysis (Fig. 7) and stage pass rates (Table 8) genuinely support that decomposition. The Eq. (1) model is also useful: held-out RMS of 5.7 points for π0.5 against a 4.0-point noise floor is real predictive power, and the authors report its failures (GR00T's negative fitted coefficients, near-static async at 250 ms) rather than burying them.\n\nSoft spots, in proportion. The stress-test note lands: the §5.2 deployment-selection claim is validated on exactly one cell — π0.5, set-down, 0.075 m/s. The four 'systems' differ only in injected latency and throughput cap over the same MuJoCo surface, so cross-task and cross-policy generality of the selection procedure is unestablished. This is the biggest gap between the abstract's 'guide the selection' claim and the evidence. Not fatal: the model itself is validated across four tasks and held-out latencies for π0.5 and MolmoAct2, so the mechanism isn't fragile, just the system-selection demonstration is thin. Note System A in Table 3 is its own calibration curve; the held-out evidence in that table is Systems B–D, plus the 133-point validation in Appendix D.\n\nSecond, the benchmark artifacts aren't released; for a benchmark paper that's a concrete omission. Third, the abstract's 5.7–7.8% range covers two of the three policies; GR00T's error under the same protocol isn't reported in the main table (its limited-calibration median in Appendix D is 11.6), so 'across various VLA policies' mildly overstates coverage. Fourth, the real-robot confirmation is five rollouts per condition on one policy — fine as a sanity check, thin as evidence. GR00T's sensitivity to initial target positions (Table 1) is a real confound for its fitted coefficients; the paper acknowledges it, to its credit, but it means the speed-scaling story should be read as π0.5/MolmoAct2-first.\n\nThis paper deserves a serious referee. The revision that would make it solid: release the code, data, and waypoints; temper the abstract; report GR00T's held-out error in the main table; and either extend §5.2 to a second task or policy or state plainly that selection is demonstrated for one cell.","headline":"Well-measured latency benchmark with a genuinely predictive empirical model; the core findings are solid, and the main weakness is that the deployment-selection claim rests on a single validation cell.","tokens_in":33444,"tokens_out":8269,"duration_ms":48248},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper claims that even 100 ms of VLA inference latency can nearly halve robot task success on moving targets, and that a two-coefficient speed-scaling model predicts the damage well enough to pick the best execution mode for any…","keywords":["VLA inference latency","robot manipulation benchmark","synchronous and asynchronous execution","observation staleness","robot idle time","speed-scaling success model","moving-object manipulation","execution horizon"],"falsifier":"Take a policy-task pair not used in the paper, measure its reference curve $S_0$, then measure success under a synchronous configuration at a latency whose scaling factor $\\lambda$ is large, at target speeds where $S_0$ falls steeply; the identity predicts the measured points sit on $S_0$ at the rescaled speeds, so a systematic miss beyond the roughly 4-point combined sampling noise of 200-rollout runs would falsify it. The paper's own data already mark where to look: at 250 ms asynchronous execution on near-static targets, measured release success drops from 93.0% to 76.0% while the model predicts a shift under 0.2 points, so reproducing that drop, or failing to, decides whether the breakdown is a corner case or the rule.","tokens_in":33937,"feed_emoji":"🤖","tokens_out":12302,"duration_ms":78760,"temperature":0.7,"pith_summary":"This paper tries to establish a quantitative answer to a question robotics system builders currently answer by guesswork: how many points of task success does VLA inference latency cost, and why. Working from four moving-target manipulation tasks in the LIBERO/MuJoCo simulator, three open VLA policies, and a controlled sweep of latencies, execution modes, and target speeds, it finds that even 100 ms of inference latency can nearly halve success on a precision task -- set-down drops from 71.5% to 40.5% at 0.1 m/s -- while a more tolerant grasping task keeps 86% of its zero-latency success. The paper attributes the loss to two separable mechanisms, stale observations and robot pauses, and shows that asynchronous execution recovers part of the damage at the price of more frequent inference requests. On top of these findings it claims that the whole latency effect can be summarized by reading a measured reference success curve at a rescaled target speed, with a two-coefficient adjustment linear in staleness and idle time; this model predicts held-out latencies within 5.7-7.8 points RMS and selects the measured-best execution mode and horizon on all four deployment systems it considers. If the claim holds, a few calibration measurements on one system could replace an expensive configuration sweep on every new GPU, network, or cloud endpoint.","feed_headline":"100 ms of AI lag nearly halves robot success on moving targets","feed_subtitle":"It traces the loss to stale views and robot pauses, and a two-coefficient model predicts the best deployment mode.","key_machinery":"The load-bearing object is Eq. (1), a speed-scaling identity: the success of configuration $X$ is the reference curve $S_0$ read at a rescaled target speed $\\lambda_X v$, where $\\lambda_X$ is linear in the relative change of mean observation staleness and in idle steps per executed action. Its function is to compress an expensive configuration search into a curve lookup: once $S_0$ and the two coefficients are measured, every combination of latency, execution mode, and execution horizon maps to one point on one curve, and the coefficients themselves become an interpretable diagnostic -- a large $a$ means the task is hurt by stale observations, a large $b$ means it is hurt by pauses. The identity also supports calibration without zero-latency data, because $S_X(v) \\approx S_R((\\lambda_X/\\lambda_R)v)$ lets any measured curve serve as the reference, provided one calibrated configuration is synchronous at nonzero latency so that idle time can be separated from staleness.","core_discovery":"On its own terms, the paper's central claim is that inference latency damages VLA task success through exactly two channels -- the observation the policy acts on is stale by the time the action executes, and the robot sits idle while inference runs -- and that their combined effect is equivalent, for the purpose of predicting success, to making the target move faster. That equivalence is formalized as $S_X(v) \\approx S_0(\\lambda_X v)$, with $\\lambda_X = 1 + a\\,(\\bar{s}_X - \\bar{s}_0)/\\bar{s}_0 + b\\,I_X/H_X$, where $S_0$ is a measured reference success curve and the two fitted coefficients $a$ and $b$ quantify a policy-task pair's sensitivity to staleness and idle time respectively. Fitted on zero-, 100-, and 200-ms runs, the model predicts held-out 50-, 150-, and 250-ms curves with 5.7 points RMS error on $\\pi 0.5$ and 7.8 on MolmoAct2, close to the 4-point sampling noise of the measurements themselves; with only one complete curve plus two points it holds a median 7.7-point error. Applied to four deployment systems differing in GPU latency and per-robot throughput, it predicts the measured best execution mode and horizon in every case. The paper is explicit about where the equivalence stops: GR00T N1.7 fits negative coefficients on some tasks, partly because changing initial target positions already costs that policy 30-40 points at zero latency, and near-static asynchronous runs at 250 ms lose success the model cannot see.","pith_inferences":["If the speed-scaling identity holds beyond the tested grid, it can be inverted into a design specification: for a target success rate and task, $S_0$ and $\\lambda$ give the maximum tolerated latency and idle-time budget, letting a systems team state a hardware requirement ('no more than X ms end-to-end') in task-success terms instead of frames-per-second terms.","The two-channel decomposition suggests a direct evaluation recipe for latency-robust training: policies trained with deliberately stale observations or inserted idle gaps should show smaller fitted $a$ and $b$, so the benchmark's coefficients could serve as the metric for such methods.","The model's acknowledged failures -- GR00T's negative coefficients and the 250 ms near-static asynchronous drop -- point to a missing third mechanism, plausibly chunk-boundary discontinuities or sensitivity to initial target position; the residuals at extreme latencies are where such a term would be identified and tested.","Because the best policy depends on task and speed as well as latency, single-number leaderboard rankings of VLA models are under-specified for real-time robotics; a latency-by-speed success matrix is closer to what a deployment decision actually needs."],"forward_implications":["For a given deployment, the best execution mode and horizon can be chosen from three or four measurements: the model matched the measured best configuration on all four representative systems, from onboard GPU to slow cloud.","Continuous asynchronous execution at short latency can beat zero-latency synchronous execution on precision tasks because it freshens observations: at 50 ms it raised set-down success at 0.125 m/s from 51.5% to 70.0%.","Latency sensitivity is not a single number per policy: it factors into task precision tolerance and motion-continuity demands, so the question of how fast inference must be has to be answered per task, per target speed, and per policy.","The two fitted coefficients per policy-task pair give a cheap screening test for which failure channel dominates, predicting whether asynchronous execution will help before a full sweep is run.","Held-out prediction error on unseen latencies (5.7-7.8 points RMS) is within about two points of the measurements' own 4-point sampling noise, so further accuracy gains would need more rollouts or better measurement rather than a richer model."],"supporting_citations":[{"why":"Supplies $\\pi 0.5$, one of the three evaluated VLA policies whose official checkpoint the benchmark runs at its default chunk size.","marker":"(Black et al., 2025a)"},{"why":"Supplies the GR00T N1.7 LIBERO checkpoints, the outlier policy whose negative fitted coefficients mark where the speed-scaling model breaks.","marker":"(NVIDIA, 2026a)"},{"why":"Supplies MolmoAct2, the third evaluated policy, which also carries the real-robot verification in Appendix F.","marker":"(Fang et al., 2026)"},{"why":"Provides LIBERO, the task suite and MuJoCo layouts from which the four benchmark tasks and their static success baselines are drawn.","marker":"(Liu et al., 2023)"},{"why":"Provides MuJoCo, the physics engine whose unmodified contact model resolves every robot-target interaction in the primary benchmark.","marker":"(Todorov et al., 2012)"},{"why":"Supplies DynamicVLA and the DOM benchmark in Isaac Sim, the secondary simulator-policy combination used to cross-check Findings 1-3.","marker":"(Xie et al., 2026)"},{"why":"Provides evidence that perturbing initial object states sharply reduces policy success, used to justify unchanged layouts and to explain GR00T's negative coefficients.","marker":"(Zhou et al., 2025)"},{"why":"Supplies the VLA-Perf latency reference points and network-latency estimates that set the latencies of the four deployment scenarios.","marker":"(Jiang et al., 2026)"},{"why":"Supplies FlashRT, the runtime whose measured GPU latencies define the inference latencies of the onboard and edge deployment systems.","marker":"(Su, 2026)"}],"fun_headline_variants":["100 ms of lag nearly halves moving-target robot success","Why 100 ms of VLA latency breaks moving-target tasks","Stale views plus idle robot time explain VLA lag costs","Two-coefficient model picks robot deployment mode from latency","Latency sensitivity predicts VLA success across policies"],"cache_read_input_tokens":0,"weakest_assumption_plain":"Everything rests on the equivalence between latency damage and a faster target: success under any configuration is read off one measured curve at a speed rescaled linearly by staleness and idle time, an equivalence the paper itself shows already breaks for GR00T N1.7 and for near-static asynchronous execution at 250 ms.","fun_headline_variants_meta":{"raw":{"variants":["100 ms of lag nearly halves moving-target robot success","Why 100 ms of VLA latency breaks moving-target tasks","Stale views plus idle robot time explain VLA lag costs","Two-coefficient model picks robot deployment mode from latency","Latency sensitivity predicts VLA success across policies"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000133,"raw_usage":{"cost":0.0001326416,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0001326416,"upstream_inference_prompt_cost":3.47256e-05,"upstream_inference_completions_cost":9.7916e-05},"total_tokens":1730,"prompt_tokens":1192,"completion_tokens":538,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":1013,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":471}},"tokens_in":1192,"tokens_out":538,"cost_source":"openrouter_usage","duration_ms":6513,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":471,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-03T17:51:44.230869+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Take a policy-task pair not used in the paper, measure its reference curve $S_0$, then measure success under a synchronous configuration at a latency whose scaling factor $\\lambda$ is large, at target speeds where $S_0$ falls steeply; the identity predicts the measured points sit on $S_0$ at the rescaled speeds, so a systematic miss beyond the roughly 4-point combined sampling noise of 200-rollout runs would falsify it. The paper's own data already mark where to look: at 250 ms asynchronous execution on near-static targets, measured release success drops from 93.0% to 76.0% while the model predicts a shift under 0.2 points, so reproducing that drop, or failing to, decides whether the breakdown is a corner case or the rule.","supporting_citations":[{"cited_title":"MolmoAct2: Action Reasoning Models for Real-world Deployment","cited_arxiv_id":"2605.02881","evidence_quote":"Supplies MolmoAct2, the third evaluated policy, which also carries the real-robot verification in Appendix F."},{"cited_title":"LIBERO : Benchmarking knowledge transfer for lifelong robot learning","cited_arxiv_id":null,"evidence_quote":"Provides LIBERO, the task suite and MuJoCo layouts from which the four benchmark tasks and their static success baselines are drawn."},{"cited_title":"MuJoCo: A physics engine for model-based control","cited_arxiv_id":null,"evidence_quote":"Provides MuJoCo, the physics engine whose unmodified contact model resolves every robot-target interaction in the primary benchmark."},{"cited_title":"DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation","cited_arxiv_id":"2601.22153","evidence_quote":"Supplies DynamicVLA and the DOM benchmark in Isaac Sim, the secondary simulator-policy combination used to cross-check Findings 1-3."},{"cited_title":"LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization","cited_arxiv_id":"2510.03827","evidence_quote":"Provides evidence that perturbing initial object states sharply reduces policy success, used to justify unchanged layouts and to explain GR00T's negative coefficients."},{"cited_title":"arXiv preprint arXiv:2602.18397 , year=","cited_arxiv_id":null,"evidence_quote":"Supplies the VLA-Perf latency reference points and network-latency estimates that set the latencies of the four deployment scenarios."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}