{"id":"876ebd97-5ee2-4794-b39e-4a6aeb319231","arxiv_id":"2610.34717","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Edge-robot AI latency is set by the number of action-generation steps and by the inference backend, not by model parameters, and after optimization only few-token stages hit the memory-bandwidth wall.","lead":"VLA-HWBench runs 18 open robot AI models on two edge computers and splits each inference call into seven timed stages. It finds that the way a model generates actions, not its size, determines latency, and that specialized fused 4-bit inference engines give the largest speedups but exist for only two models on one platform.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Insight 3's memory-bound labels rest on derived, not measured, per-step weight traffic; a ~40% byte overestimate would push the denoising steps from 1.6-1.8x to ~2.7-3.0x above the roofline and flip them to software-limited.","rationale":"This is a careful measurement paper with genuine internal discipline: stage sums checked against totals, per-step sweeps with R² ≥ 0.986 for every reference configuration, disclosed action changes under compilation (Table 14), frame-to-frame variation reported, and the FlashRT analysis explicitly scoped to two models on one platform. Insights 1 and 2 have multiple independent supports: the same-weights MolmoAct2 head comparison, the iteration sweeps, the R² ≤ 15% parameter regression, and the cosine checks on the precision steps. The load-bearing seam is Insight 3, which is also the piece with the longest evidence chain: F and W derived from eager-run GEMM shapes in optimized formats, an assumption about a third-party runtime's per-step byte reads (KV and modulation cached once per call), and per-step times from sweeps and graph-kernel attribution. The decisive weakness is directional asymmetry: if FlashRT reads more bytes than assumed the memory-bound label tightens; if it reads fewer (fused or sparsified kernels, format conversion effects), the stages move away from the roofline and the label flips — a ~40% byte overestimate is enough to push the 1.6-1.8x stages to ~2.7-3.0x and erase the boundary Insight 3 draws. The protocol cannot detect this because it never measures FlashRT's actual traffic. I agree with the reader's weakest_assumption; my refinement is that the decisive quantity is measured traffic versus derived W, and that the GR00T FlashRT step sweep (R² = 0.868) makes the time side of the ratio the paper's least certain fitted quantity. Verdict UNCHANGED: the concern is genuine, concrete, and cheaply testable, but it is an addressable verification gap rather than a demonstrated error; Insights 1 and 2 stand independently, and even a large byte error leaves the few-token stages the most bandwidth-proximal stages on the Jetson. The paper's explicit disclosure of the assumption (Appendix G) and its honest scoping support keeping the conditional acceptance posture.","tokens_in":25279,"tokens_out":32368,"duration_ms":279252,"concrete_test":"On the Jetson, run the fastest FlashRT NVFP4 configurations of π0.5 and GR00T N1.7 under CUPTI and read the actual DRAM bytes transferred per denoising step (ncu as a fallback, mindful of its replay effects). Compare measured per-step bytes against Table 15's W columns (322 MB for π0.5 action, 489 MB for GR00T) and measured per-step kernel time against the 1.92 ms and 3.30 ms used there. Separately, re-sweep the GR00T N1.7 FlashRT step count with more points: Table 10 reports R² = 0.868 for that sweep, far below every other sweep in the paper. If measured per-step bytes are within ±20% of W and the re-swept slope within ±20% of the reported step time, the 1.6-1.8x distance to the memory bound and the 'memory-bound / hardware-limited' label for action generation stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Insight 3 and its hardware corollary — after optimization only the few-token stages are memory-bound, so edge hardware should prioritize bandwidth and a fast host CPU over peak compute — rest entirely on the Appendix G / Table 15 accounting: F and W come from GEMM shapes recorded in an eager run, converted into each optimized format, and a FlashRT denoising step is assumed to read exactly those W bytes once per step, with modulation and KV context cached once per call. The per-step byte count is derived, not measured; FlashRT's actual DRAM traffic is never observed. The error direction is not neutral. If FlashRT's fused kernels merge, skip, or sparsify weight blocks, or its NVFP4 path loads fewer bytes than the converted eager shapes imply, then W/B shrinks, the measured step time sits farther above the roofline than the reported 1.6-1.8x, and the 'hardware-limited' label flips toward 'software-limited'. Quantitatively, a ~40% overestimate of W pushes the three Jetson action and prefill stages from 1.6-1.8x to ~2.7-3.0x above the bound, erasing the boundary Insight 3 draws and removing the direct evidence that adding memory bandwidth accelerates the bottleneck stage. This is exactly the direction the protocol cannot see: Appendix G states the once-per-step caching assumption explicitly (credit where due), but neither the eager-shape accounting nor the kernel-order graph attribution would reveal fewer actual bytes. Two stresses compound this: the GR00T N1.7 FlashRT step sweep has R² = 0.868 (Table 10), far below every other sweep in the paper, so the time side of the ratio is itself the least certain fitted quantity; and the graph-stage attribution splits captured CUDA graphs by kernel order and repetition counts, plausible but unverified for FlashRT's fused graphs. The opposite error (more bytes, e.g., re-reading KV per step) would only tighten the memory-bound label; the uncontrolled direction is under-reading of W.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces VLA-HWBench, a benchmark that measures the end-to-end latency of 18 open VLA models (19 reference configurations) on an NVIDIA Jetson AGX Thor and an AMD Ryzen AI Max+ PRO 395, with a seven-stage breakdown of each inference call. It reports three insights: action-generation strategy, not model size, dominates latency; a fused-kernel backend with 4-bit weights gives the largest speedups but exists for only two models on the Jetson; and after optimization only few-token stages such as denoising steps are memory-bound on the Jetson, while vision encoding, prefill, and CPU preprocessing are software-limited. The paper concludes with hardware, software, and model-design recommendations: reduce action-generation iterations, extend fused low-bit backends, and prioritize memory bandwidth and host CPU speed over peak GPU compute.","tokens_in":25715,"tokens_out":25153,"duration_ms":200064,"significance":"If correct, this is a valuable benchmark for the edge-VLA community: it is substantially broader in model coverage and stage granularity than prior studies, and it gives concrete, falsifiable guidance for model, runtime, and hardware co-design. The methodology is unusually careful in several respects: end-to-end latencies are reported with 5th-95th percentile spreads, stage times are checked against end-to-end time, per-step costs come from swept linear fits with mostly very high R2, and the paper discloses action changes under compilation and the existence of fast reasoning modes that change task behavior. The two-platform design also usefully separates host, GPU, and memory effects. The main risk is that the bottleneck classification underlying Insight 3 rests on derived, rather than measured, memory traffic for the FlashRT configurations, and the central stage-count claim in Insight 1 needs to be reconciled with the printed stage tables.","major_comments":[{"comment":"The memory-bound versus software-limited classification of the FlashRT Jetson stages rests on derived, not measured, per-step memory traffic. Appendix G states that F and W are taken from GEMM shapes recorded in an eager run and converted into each optimized format, and that a FlashRT denoising step reads only the weights of its GEMMs because modulation and KV context are cached once per call. No profiler measurement of FlashRT's actual DRAM bytes is reported. This is load-bearing: if FlashRT's fused kernels read fewer bytes per step than the converted eager shapes imply, the 1.6x (pi0.5 action), 1.7x (GR00T prefill), and 1.8x (GR00T action) distances to the W/B bound would grow; a 40% reduction in W would put them at roughly 2.7-3.0x above the bound, in the same range as the 2.8-3.3x AMD values that the paper labels software-limited. Since the hardware recommendation (bandwidth and host CPU over peak compute) follows from this distinction, I ask for direct validation of FlashRT's memory traffic, for example with hardware memory-throughput counters, or for an explicitly conditional formulation of Insight 3.","section":"Section 5.3, Insight 3; Appendix G; Table 15","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"As printed, the stage tables do not reproduce the claim that action generation is the largest stage in 14 of the 19 configurations. Under the arrow convention stated in Appendix C, X-VLA shows S3=51.7 ms and S4=0.1 ms, and RIPT-VLA shows S3=223.6 ms and S4=99.5 ms; taken together with VLA-Adapter, SimpleVLA-RL, OpenVLA-OFT, and Evo-1, these rows would make action generation the largest stage in 13 configurations rather than 14. If X-VLA's flow-matching action generation is executed inside the call timed as prefill and is therefore counted in S3, the table and the text should state this explicitly. Otherwise the count or the table needs correction before the paper's first central claim can be checked by a reader.","section":"Section 5.1, Insight 1; Tables 8-9","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The arrow notation indicating fused stages is not self-explanatory: a reader cannot tell whether the arrowed source time is included in the target stage's total or listed separately. Please define the notation in the caption and show the resulting column sums explicitly.","section":"Tables 8-9","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Several rows contain negative S6 residuals (for example, -3.0 ms and -1.1 ms). Since the paper states that stage sums are checked against end-to-end time, a brief explanation of these negative residuals as timing jitter or rounding would remove a confusing discrepancy.","section":"Tables 8-9 and Section 4.4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The AMD-platform circles do not show the 5th-95th percentile spreads even though Table 7 reports them; adding whiskers or a note explaining why they are omitted would improve comparability with the Jetson bars.","section":"Figure 4a","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper says \"one point per checkpoint\" for the parameter-count regression, but MolmoAct2 contributes two reference configurations; please clarify whether the regression uses 18 or 19 points and whether the two MolmoAct2 points are treated as independent.","section":"Table 2 and Figure 1","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Short version: this is a careful, useful benchmark paper and it deserves a serious referee. The first two insights hold up; the third one is shakier than the prose lets on.\n\nWhat is genuinely new: 18 open VLAs, both edge platforms, all four action-generation forms, and a seven-stage breakdown that finally times CPU preprocessing. Prior studies covered at most five models and two forms. The methodology is unusually disciplined: stage sums are checked against end-to-end time, step costs come from swept linear fits with R2 above 0.99 almost everywhere, latency is reported with 5th–95th percentiles, and the paper discloses that compilation changes actions, including a flipped gripper command for pi0.5. That level of transparency earns real credit. Insight 1 — action-generation iterations, not parameter count, set latency — survives reading: the 82x and 157x spreads are measured numbers, and the log-parameter-count fit explains at most 15 percent of variance.\n\nThe soft spots, in proportion. Insight 3 is the fragile one. Appendix G states the assumption honestly: F and W come from GEMM shapes recorded in an eager run, converted into each optimized format, and a FlashRT denoising step is assumed to read exactly those W bytes, with modulation and KV context cached once per call. But no DRAM traffic is measured. The stress-test arithmetic lands: a roughly 40 percent overestimate of W moves the Jetson action stages from 1.6–1.8x to about 2.7–3.0x above the memory bound, which erases the boundary Insight 3 draws between memory-bound and software-limited. The time side of that ratio is also the weakest fit in the paper: the GR00T N1.7 FlashRT step sweep has R2 = 0.868, far below the others. The hardware recommendation — prioritize bandwidth and a fast host CPU over peak compute — is plausible but unverified. I would not build a bandwidth-first edge board on it yet.\n\nSecond: no benchmark artifacts are released or linked. No code, data, or traces. For a benchmark whose stated purpose is to be extended, that is a real gap. Third: the LeRobot-to-FlashRT swap is the largest single gain in the paper (5.6x over eager, 2.15x over compiled), and it is the one optimization missing from the action-agreement table. The authors clearly know how to do this check; its absence for the flashiest number is a hole.\n\nWho this is for: edge-inference and robot-systems people will get real value, and the benchmark itself is a contribution even if Insight 3 needs tightening. Send it to a serious referee, and push for artifacts, the missing action check, and either a measured byte count or a softened Insight 3.","headline":"A genuinely useful measurement paper with a robust first two insights, a fragile third insight that rests on unmeasured byte counts, and one missing action-equivalence check on its flashiest number.","tokens_in":26382,"tokens_out":2960,"duration_ms":38103},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper claims that a robot model's inference latency on edge hardware is set by its action-generation method, runtime backend and platform — not by its model size — across 18 open models measured stage by stage on two chips.","keywords":["vision-language-action models","edge inference latency","action generation","flow matching","parallel decoding","kernel fusion","low-bit quantization","roofline analysis"],"falsifier":"Trace the Jetson's DRAM traffic during one FlashRT denoising step and compare actual bytes moved with the weight bytes $W$ assumed in the Appendix G table; if real per-step traffic exceeds $W$ substantially, or if halving weight bytes via a lower format fails to shrink step time near the memory-bound slope, the memory-bound label for the action stage is wrong. A second check: run the optimized configurations on a board with matched compute but doubled memory bandwidth, since the paper's diagnosis predicts the action stages shrink toward the bound while vision encoding and prefill stay roughly flat.","tokens_in":26643,"feed_emoji":"🤖","tokens_out":15250,"duration_ms":83572,"temperature":0.7,"pith_summary":"The paper sets out to show where inference time actually goes when vision-language-action (VLA) robot models run on edge hardware, and which levers — model design, inference software, or hardware — actually reduce it. It builds VLA-HWBench, which splits one inference call into seven timed stages and measures 18 open VLA models on two architecturally different edge platforms, an NVIDIA Jetson AGX Thor and an AMD Ryzen AI Max+ PRO 395. The central result is that latency follows the number of sequential iterations a model spends generating actions — denoising steps, decoded tokens, reasoning tokens — rather than its size: latency spans $82\\times$ on the Jetson and $157\\times$ on AMD, and parameter count explains at most 15% of the variance. The paper also shows that fused-kernel, 4-bit-weight backends give the largest single speedup, 2.6–2.9$\\times$ over PyTorch with torch.compile, but exist for only two models on one platform, and that after optimization memory bandwidth limits only the few-token stages such as denoising, while vision encoding, prefill, and CPU preprocessing are software-limited. If these claims hold, edge VLA designers should cut action-generation iterations before shrinking models, inference engines should adopt fused low-bit backends, and edge hardware should buy memory bandwidth and host CPU speed ahead of peak compute.","feed_headline":"Edge robot latency follows action method, not model size","feed_subtitle":"An 18-model, two-chip benchmark shows latency tracks action-generation iterations; fused 4-bit kernels win.","key_machinery":"The load-bearing machinery is a seven-stage decomposition of one observation-to-action call, $T = \\sum_{k=0}^{6} T_k$, whose stages are CPU preprocessing (S0), host-to-device copy (S1), vision encoding (S2), language-model prefill (S3), action generation (S4), postprocessing (S5), and an S6 residual; stages are timed at the boundaries of each release's own code and split further by kernels inside CUDA graphs. The central sub-identity is $T_4 \\approx N t$, which says action-generation time equals the number of iterations $N$ — denoising steps for flow-matching heads, one pass for parallel decoders, generated tokens for autoregressive heads, reasoning tokens plus action iterations for reasoning models — times the per-iteration cost $t$. To classify what limits each stage, the paper applies the roofline model, computing each stage's arithmetic intensity $F/W$ against the ridge point $P/B$ and comparing achieved time with the compute bound $F/P$ and the memory bound $W/B$; a stage within a small factor of its bound is hardware-limited and one far from it is software-limited. The two platforms are chosen so that peak compute differs 4.4$\\times$ while memory bandwidth differs only 1.1$\\times$, which separates host, GPU, and memory effects.","core_discovery":"Stated in the paper's own terms, the discovery is that a VLA's end-to-end inference latency on edge hardware is governed jointly by its action-generation method, its runtime backend, and the platform, not by its parameter count. Across 19 reference configurations of 18 released models, action generation is the largest of the seven stages in 14 of 19 cases on the Jetson, taking 42–99% of the call, and median latency ranges from 77 ms to 6.3 s on the Jetson ($82\\times$) and from 114 ms to 17.9 s on AMD ($157\\times$); a log-log fit of latency against loaded parameters explains at most 15% of the variance, and the gap is accounted for by the identity $T_4 \\approx N t$, in which $N$ is the number of action-generation iterations. The paper further reports that stacking compilation, a specialized fused backend, and 8-bit or 4-bit weights makes $\\pi_{0.5}$ 7.2$\\times$ faster and GR00T N1.7 2.6$\\times$ faster on the Jetson, with the fused 4-bit backend alone worth 2.6–2.9$\\times$ over PyTorch with torch.compile, and that this optimization pushes time into the stages it shortens least, so CPU preprocessing grows from 12% to 39% of the GR00T N1.7 call. A roofline analysis of the fastest configurations then finds the few-token action steps memory-bound on the Jetson, running 1.6–1.8$\\times$ above the weight-read bound, while vision encoding, prefill, and host preprocessing sit far from their hardware limits and are therefore software-limited.","pith_inferences":["Extension the authors leave implicit: the same recipe — count a model's sequential action-generation iterations and time one iteration — could serve as a cheap pre-deployment screen for any new VLA, because it predicts the latency ordering of the 19 configurations without building new hardware or software.","The paper's roofline tables suggest a portability test it did not run: whether FlashRT-style fused kernels on the AMD platform would close the 2.8–3.3$\\times$ gap its action steps show below the memory roof, with Isaac-GR00T's per-step recomputation of keys and values the first suspect.","If the memory-bound diagnosis of denoising steps is right, the next generation of action heads should be designed to shrink weight bytes per step — lower-bit weights, shared modulation — rather than to raise compute throughput, a bias that would show up as speedups scaling with byte reduction.","A testable prediction beyond the paper: on a board with the same compute but double the memory bandwidth, the optimized Jetson action stages should shrink toward the memory bound while vision encoding and prefill stay roughly flat, cleanly separating the hardware and software limits the paper labels."],"forward_implications":["VLA design for edge robots should cut action-generation iterations — fewer denoising steps, parallel decoding, or longer action chunks — before shrinking parameters, because only 10 of 19 Jetson configurations keep up with LIBERO's 20 Hz control rate and the $82$–$157\\times$ latency spread tracks iteration counts, not size.","Inference software should port fused-kernel, low-bit-weight backends beyond $\\pi_{0.5}$ and GR00T N1.7 and beyond the Jetson, since that single change is worth 2.6–2.9$\\times$ over torch.compile, and model code should compile as one graph so the whole call is captured.","Edge hardware should be specified for memory bandwidth and host CPU throughput ahead of peak matrix compute, because after optimization the few-token action stages are memory-bound and CPU preprocessing is the largest untouched stage, at 39% of the GR00T N1.7 call.","Latency engineering must treat host preprocessing and GPU kernels as one system, because every optimization shortens GPU stages unevenly and simply moves the remaining time into the stages it shortens least."],"supporting_citations":[{"why":"Supplies the LIBERO-10 observation every configuration runs and the 20 Hz control rate that defines whether a configuration keeps up with the robot controller.","marker":"(Liu et al., 2023)"},{"why":"The π0.5 release, one of the two models on which the full optimization ladder (compile, backend, precision) is measured.","marker":"(Black et al., 2025a)"},{"why":"GR00T N1, the other optimized model; its released code is the reference configuration and its compile recipe is measured as a baseline.","marker":"(Bjorck et al., 2025)"},{"why":"FlashRT, the fused-kernel backend whose FP8 and NVFP4 configurations deliver the largest speedups and anchor the memory-bound analysis.","marker":"(Su, 2026)"},{"why":"TensorRT, the alternative specialized backend whose smaller GR00T N1.7 gain is attributed to keeping the vision encoder in FP32.","marker":"(NVIDIA, 2026)"},{"why":"Roofline model, the analytical tool used to classify every GPU stage as memory-bound or compute-bound and to label it hardware- or software-limited.","marker":"(Williams et al., 2009)"},{"why":"PyTorch 2 compilation; torch.compile is the automatic-optimization baseline that every specialized backend is compared against.","marker":"(Ansel et al., 2024)"},{"why":"Prior profiling study attributing up to 75% of edge VLA latency to action generation, the closest earlier result that VLA-HWBench extends and checks.","marker":"(Vishwanathan et al., 2026)"}],"fun_headline_variants":["Latency follows action generation, not model size","Action steps, not parameter count, set edge VLA latency","Edge VLA delay hinges on action-generation strategy","Fused 4-bit kernels speed two robot VLA models 2.6x","Robot VLA latency: action method beats model size"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The memory-bound versus software-bound classification rests on an accounting that assumes FlashRT's kernels read only their GEMM weights once per denoising step, with modulation and key/value context computed once per call; if the real kernels move different byte counts per step, the distance-to-bound figures and the hardware recommendations built on them would shift.","fun_headline_variants_meta":{"raw":{"variants":["Latency follows action generation, not model size","Action steps, not parameter count, set edge VLA latency","Edge VLA delay hinges on action-generation strategy","Fused 4-bit kernels speed two robot VLA models 2.6x","Robot VLA latency: action method beats model size"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":9.1e-05,"raw_usage":{"cost":9.09636e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":9.09636e-05,"upstream_inference_prompt_cost":4.01856e-05,"upstream_inference_completions_cost":5.0778e-05},"total_tokens":1659,"prompt_tokens":1380,"completion_tokens":279,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":1173,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":209}},"tokens_in":1380,"tokens_out":279,"cost_source":"openrouter_usage","duration_ms":3768,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":209,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-28T02:07:56.146830+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Trace the Jetson's DRAM traffic during one FlashRT denoising step and compare actual bytes moved with the weight bytes $W$ assumed in the Appendix G table; if real per-step traffic exceeds $W$ substantially, or if halving weight bytes via a lower format fails to shrink step time near the memory-bound slope, the memory-bound label for the action stage is wrong. A second check: run the optimized configurations on a board with matched compute but doubled memory bandwidth, since the paper's diagnosis predicts the action stages shrink toward the bound while vision encoding and prefill stay roughly flat.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":2}