{"id":"ed841f97-121a-4dd3-86b0-fb2fca0609a4","arxiv_id":"2608.14379","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new benchmark for dynamic, latency-sensitive robot manipulation, plus a compact VLA model combining latent future prediction, temporal fusion, and latency optimization that raises average success on that benchmark.","lead":"This paper introduces ReflexBench, a set of six robot tasks that test how quickly and accurately vision-language-action models can respond to moving objects, with a latency-aware evaluation protocol. It also presents ReflexVLA, a compact 1B-parameter model that predicts near-future visual states, fuses multiple frames, and uses CUDA Graph replay to improve reaction-critical manipulation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main ReflexBench results appear to use a fixed 30 Hz inference frequency without reporting RTF-injected per-policy latencies; the 0.2 pp average edge over PUMA is therefore not demonstrably a real-world reaction-critical improvement.","rationale":"The reader's weakest assumption points to the fidelity of the RTF-based latency conversion and the synchronous/asynchronous protocols. My stress-test narrows this to a specific internal-consistency gap: the main experimental setup does not report whether per-policy latencies were actually converted and injected, and instead appears to use a fixed 30 Hz inference frequency. This is load-bearing because the central claim—that a compact model outperforms larger VLAs on reaction-critical tasks—depends on the benchmark faithfully reproducing real-world latency effects. If the fixed-frequency protocol was used, slower baselines receive an unrealistic advantage, and the 0.2 percentage point margin over PUMA is meaningless. However, this is a missing-detail concern rather than a demonstrated error: the paper's internal ablations and real-world experiments are suggestive, and the authors could satisfy the concern by providing the protocol details and rerunning the comparison. The reader's CONDITIONAL verdict already requests per-policy latency reporting and artifact release, so no verdict change is needed; the condition should explicitly include the RTF-injection check. I do not see grounds for REJECT because the paper's own data do not contradict the claim, and the missing information could plausibly confirm it. I also credit the progressive ablation (Table III) and the real-world experiments as evidence that the design direction has merit, even though the main simulation claim is currently under-specified.","tokens_in":10004,"tokens_out":12006,"duration_ms":105759,"concrete_test":"Request the authors to release per-policy end-to-end latency measurements, RTF values per task, and the exact simulation-desynchronization script, then rerun Table I in two modes: (a) with RTF-injected per-policy latency as described in Section III-C, and (b) with a fixed 30 Hz frequency. If the ReflexVLA versus PUMA average margin differs by more than 3 percentage points between modes, or if the ranking flips, the headline claim is not robust. Additionally, report standard deviations for the six-task average to test whether 50.4% versus 50.2% is statistically significant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C motivates a latency-aware protocol in which real-world policy latency is converted to a simulation delay via RTF = t_sim/t_wall (Eq. 1), with synchronous and asynchronous blocking. Yet the main experiments (Section V-A) specify only that all policies use asynchronous inference, chunk size 8, horizon 2, on one RTX 5880 Ada GPU; Figure 5 fixes inference frequency at 30 Hz. No per-policy end-to-end latencies, no RTF values, and no statement that RTF-converted delays were actually injected are reported for Table I. If the protocol instead runs every model at a prescribed 30 Hz, then slower models such as PUMA and OpenVLA are granted a control frequency they could not physically sustain, breaking the realism the benchmark claims. The headline margin is also fragile: ReflexVLA's 50.4% average exceeds PUMA's 50.2% by 0.2 percentage points, and ReflexVLA trails PUMA on Rolling Ball Interception (77.1 vs 85.1) and Ball Throwing (31.7 vs 33.8), so the abstract's 'consistently improves' is not supported by per-task data. The missing protocol details make the central claim unverifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ReflexBench, a benchmark of six simulated reaction-critical manipulation tasks with a latency-aware evaluation protocol that decouples simulation stepping from robot control under synchronous and asynchronous inference, including an RTF-based conversion from wall-clock policy latency to simulation delay. Building on ReflexBench, the authors propose ReflexVLA, a 1B-parameter vision-language-action model that adds latent future prediction with a frozen DINOv3 target, multi-frame temporal fusion in the vision backbone, and inference-time optimizations (batched visual encoding and CUDA Graph replay). Experiments report a 50.4% average success rate on ReflexBench, outperforming all compared baselines, a 97.2% average on LIBERO, and real-world validation on three tasks.","tokens_in":10204,"tokens_out":3185,"duration_ms":29158,"significance":"If the results hold, the contribution is valuable: it addresses a real gap in VLA evaluation by making latency an explicit part of the benchmark, and it shows that a compact 1B model can compete with much larger models on dynamic manipulation without large-scale robot-data pretraining. The paper also contains useful empirical studies of how inference frequency, chunk size, and action horizon affect success in dynamic tasks, and the progressive ablation with measured latency gives insight into the contribution of each component. However, the strength of the central claim depends on the fairness and faithful implementation of the benchmark protocol, which is not fully documented, and on the representativeness of a benchmark built entirely by the same group that proposes the method. The thin 0.2 pp average margin over PUMA and the absence of error bars in the ablation further limit confidence. The paper includes machine-checked experimental comparisons in the sense that tables are internally consistent, but no code or data is released, so reproducibility cannot be independently verified.","major_comments":[{"comment":"The main ReflexBench comparison does not report per-policy end-to-end latencies, RTF values, or any statement that the RTF-converted delays of Eq. (1) were actually injected into the evaluation. Section V-A states only that all policies use asynchronous inference with chunk size 8, horizon 2, on one RTX 5880 Ada GPU, and Figure 5 fixes the inference frequency at 30 Hz. If every model is simply run at a prescribed 30 Hz, then slower models such as PUMA and OpenVLA are evaluated at a control frequency they could not physically sustain, which undermines the benchmark's claim of realistic latency modeling. The resulting 0.2 percentage point average advantage over PUMA (50.4% vs 50.2%) is therefore not demonstrably a real-world improvement. This issue is load-bearing for the abstract's claim of 'consistently improves' and must be resolved by reporting the actual latency protocol used for Table I, including per-policy latency measurements and the exact delay injection procedure.","section":"V-A, Table I, Section III-C"},{"comment":"The claim that ReflexVLA 'consistently improves dynamic manipulation performance' is not supported by the per-task data in Table I. ReflexVLA trails PUMA on Rolling Ball Interception (77.1 vs 85.1) and Ball Throwing (31.7 vs 33.8), and underperforms DynamicVLA on Rotating Peg Insertion (12.4 vs 16.7) and Ball Throwing (31.7 vs 36.2). The only large gains over PUMA are on Conveyor Belt Pick-and-Place (73.8 vs 67.4) and Ball Catching (7.3 vs 4.0), both of which are low or noisy. The authors should either qualify the claim to 'higher average success rate' or provide statistical evidence that the average difference is robust given the reported standard deviations (the 50.4 vs 50.2 difference is within one standard deviation of most rows).","section":"Table I, per-task rows"},{"comment":"The benchmark is self-referential: the tasks, demonstrations, and evaluation choices in ReflexBench are all created by the same group that proposes ReflexVLA, and the demonstration pipeline described in Section III-B uses a planner that predicts future trajectories from real-time position and velocity, plus task-specific RL policies for complex interactions. This pipeline may systematically encode exactly the future-tracking and multi-frame behavior that ReflexVLA is designed to exploit, while baseline models are trained on demonstrations that may not expose the same inductive bias. The paper provides no check for this, such as evaluating all methods on an existing independently-created dynamic benchmark (e.g., those in [4], [5], or [6]). Because the central claim is that ReflexVLA outperforms existing VLA approaches on reaction-critical manipulation, the absence of such a comparison is a correctness risk that should be addressed, at minimum by discussing the potential bias and by adding a cross-benchmark evaluation if feasible.","section":"III-B, V-A"},{"comment":"The progressive ablation in Table III reports only single success-rate values with no standard deviations, no number of trials, and no explicit statement of which task(s) the results correspond to (the text suggests Conveyor Belt Pick-and-Place, but the table header is ambiguous). The claimed jump from 36.8% to 62.8% from switching to a frozen future-prediction target, and from 71.7% to 73.8% from latency optimization, cannot be assessed statistically. Given that the paper emphasizes careful measurement in other places, the ablation should include at least the same three-seed mean±std protocol used in Table I, or state clearly that only one task is considered and justify that choice.","section":"Table III"}],"minor_comments":[{"comment":"There are several typographical issues, including 'we presentReflexBench' (missing space) in the abstract and 'a 1B-parameter model' being described as 'Reflex' in the title while the model is called ReflexVLA; please harmonize the naming throughout.","section":"Abstract and Section I"},{"comment":"Figure 5 heatmaps would benefit from a colorbar and explicit axis tick labels, since the numerical values inside cells are difficult to read and the color scale is not defined.","section":"Figures 4 and 5"},{"comment":"The paper does not state how the 150 evaluation episodes per task are generated (initial state distributions, random seeds, whether the same episodes are used for all models), which is needed to interpret the reported standard deviations.","section":"Section V-A"},{"comment":"The LIBERO results are reported as point accuracies without standard deviations or number of runs; while this is common in the literature, the authors should specify the evaluation protocol (e.g., one rollout per task per seed) to allow comparison with their own three-seed protocol on ReflexBench.","section":"Table II"},{"comment":"The notation G(x_text, x_vision, x_state) is used for the CUDA Graph, but the arguments are not defined in the text; please define x_text, x_vision, and x_state explicitly.","section":"Section IV-C, Eq. (11)"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on Reflex (2608.14379). The strongest contribution is ReflexBench itself: a six-task dynamic manipulation benchmark that decouples simulator stepping from policy control and models synchronous/asynchronous inference with configurable latency. That is a real gap in the current evaluation culture, and the protocol is thoughtfully described. I also think the model is a reasonable first pass at combining latent future prediction, multi-frame temporal fusion, and system-level latency optimization. The internal evidence supports the central claim: ReflexVLA improves average success on ReflexBench over its VLA-Adapter backbone from 30.3% to 50.4%, and it stays competitive on LIBERO (97.2%). Those are new numbers, not fitted constants, and the paper does not oversell the model size story.\n\nThat said, the soft spots are real and they matter. The stress-test note captures the biggest one: the main experiments (Table I) report only that all policies run asynchronously at a fixed 30 Hz, with chunk size 8 and horizon 2. There is no per-policy end-to-end latency, no RTF value, and no statement that RTF-converted delays were actually injected. If the benchmark's signature feature is the latency-aware protocol, the headline comparison needs to show it was used. Right now it reads as a fixed-frequency async eval, which is weaker and not demonstrably more realistic than what others do.\n\nThe second issue is overclaiming. ReflexVLA's 50.4% average beats PUMA's 50.2% by 0.2 points, and PUMA wins on Rolling Ball Interception (85.1 vs 77.1) and Ball Throwing (33.8 vs 31.7). The abstract's \"consistently improves\" is not supported by per-task data. The margin is fragile, and the conclusion should be phrased as \"comparable or slightly better on average,\" not \"consistently better.\"\n\nThird, no code or data are released, and there is no evaluation on existing dynamic benchmarks (e.g., the ones cited as [4], [5], [6]). That makes external validation impossible right now, especially for a self-authored benchmark. The ablation also lacks error bars, and the trainable-future-prediction variant collapsing to 4.9% success is a detail that needs more explanation than one sentence about unstable supervision.\n\nNone of this is fatal. The paper is methodologically coherent, the internal comparisons are consistent, and the limitations section is honest about the fine-tuning-only scope. I would happily referee this. But I would ask the authors to report the actual latency protocol used, release artifacts, soften the consistency claim, and add at least one external dynamic benchmark. Who is this for? Researchers working on VLA efficiency and reactive manipulation, and anyone who cares about honest latency-aware evaluation. Worth a reading group slot when the code is out, and worth citing once the protocol details are clarified.\n\nMy recommendation: send to peer review, but treat the current version as conditional on the authors addressing the protocol transparency and the overclaim.","headline":"A genuinely useful latency-aware benchmark and a plausible compact VLA, but the main result's margin is thin and the latency protocol is under-specified in the key experiments.","tokens_in":10773,"tokens_out":2060,"duration_ms":19752},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 1B-parameter robot model beats 4B and 7B rivals on reaction-critical tasks","keywords":["vision-language-action models","reaction-critical manipulation","dynamic manipulation benchmark","latency-aware evaluation","latent future prediction","multi-frame temporal fusion","CUDA graph replay","robotic manipulation"],"falsifier":"Run the same policies on a physical conveyor or ball-catching setup with a measured wall-clock action cycle, then check whether ReflexBench success rates under Real-Time-Factor-matched latency rank the policies the same way; a reversed ranking, or a failure of the injected delay to reproduce real timing sensitivity, would falsify the claim that the benchmark faithfully evaluates reaction-critical manipulation.","tokens_in":9741,"feed_emoji":"🤖","tokens_out":11644,"duration_ms":94818,"temperature":0.7,"pith_summary":"This paper argues that fast, reaction-critical manipulation is a distinct capability that small vision-language-action models can acquire without large-scale robot-data pretraining, and it provides both a benchmark and a model to demonstrate that claim. ReflexBench measures six dynamic tasks under simulated inference latency, with the simulated world continuing to move while the policy computes. ReflexVLA keeps the language model's visual-token budget unchanged by fusing multiple historical frames inside the vision encoder, and it is trained to predict future visual states in a frozen semantic feature space. The paper reports a 50.4% average success rate on ReflexBench, higher than all compared baselines, while retaining 97.2% average accuracy on the static LIBERO benchmark. The broader point is that anticipation and low latency can matter more than parameter count for dynamic manipulation.","feed_headline":"1B robot model beats 4B and 7B rivals on reaction tasks","feed_subtitle":"Future-state prediction plus low latency gives 50.4% average success on six reaction-critical manipulation tasks.","key_machinery":"The load-bearing mechanism is a masked cosine-similarity loss that makes the hidden states of learnable future tokens match features extracted by a frozen DINOv3 encoder from observations one action-chunk ahead, pulling the model's internal representations toward anticipating scene dynamics without adding inference cost. Around that core sit two supporting mechanisms: causal temporal attention over patch trajectories in the vision backbone's intermediate layer, which injects motion history without increasing the number of visual tokens the language model sees, and deployment-time optimization that batches all view-frame images into one visual-encoder call and replays the entire policy as a captured CUDA Graph, reducing end-to-end latency. ReflexBench itself is the third mechanism: it converts real inference latency into simulation delay through the Real-Time Factor and offers synchronous and asynchronous execution protocols, making latency a controllable experimental variable rather than an ignored constant.","core_discovery":"ReflexVLA's central claim is that a compact 1B-parameter policy can outperform much larger vision-language-action models on reaction-critical manipulation by combining an auxiliary objective that predicts future visual states as frozen DINOv3 features, temporal fusion across a short history applied to intermediate visual features, and system-level latency reduction through batched visual encoding and CUDA Graph replay. The paper reports that this combination raises the ReflexBench average success rate from 30.3% for its backbone model to 50.4%, exceeding all evaluated baselines, while LIBERO accuracy stays at 97.2%. The benchmark contribution is equally central: ReflexBench decouples simulation stepping from policy inference and injects configurable latency under synchronous and asynchronous protocols, so reported success rates reflect how delay and action staleness actually degrade performance. The reported ablations attribute the largest gains to frozen future targets, intermediate-layer temporal fusion, and latency optimization, with the latter cutting measured inference latency roughly in half on the flagship task.","pith_inferences":["The Real-Time Factor conversion assumes a stable ratio between simulator time and wall-clock time; a natural extension would sample latency from a distribution under variable system load, which could reorder the ranking of policies whose tolerance for timing jitter differs.","Because ReflexVLA's future tokens align one-to-one with action-chunk steps, the same auxiliary loss could be extended to variable prediction horizons or multi-step returns, potentially covering tasks with longer predictive dependencies than the six ReflexBench tasks.","Temporal fusion is applied per spatial patch across frames; extending the same causal attention across camera views as well as time could improve multi-view consistency in cluttered scenes without adding language-model tokens.","Real-world validation covered three tasks; a stronger test would measure whether ReflexBench success-rate gaps predict real-world success-rate gaps across a broader task set, since the benchmark's injection of latency is only a model of deployment conditions."],"forward_implications":["If the 50.4% average holds up, model scale is not the binding constraint for reactive manipulation: a 1B model with temporal reasoning and low latency can outperform 4B and 7B generalist VLAs.","Because frozen future targets succeed where trainable targets collapse (62.8% vs 4.9% success in the ablation), future-prediction supervision for robot policies should target a fixed semantic representation space rather than be learned end-to-end.","The latency optimizations add success on the conveyor task while roughly halving inference time (125.1 ms to 65.0 ms), implying deployment overhead should be a controlled variable in every dynamic-manipulation evaluation.","Temporal fusion on intermediate visual features outperforms fusion on final-layer features and cross-attention fusion, suggesting motion-relevant information is concentrated in mid-level visual representations.","The 97.2% LIBERO result indicates the same recipe can be added to a generalist VLA without sacrificing static benchmark performance."],"supporting_citations":[{"why":"VLA-Adapter is the backbone ReflexVLA extends; its 30.3% ReflexBench average is the direct baseline ReflexVLA raises to 50.4%.","marker":"[10]"},{"why":"The frozen DINOv3 encoder supplies the semantic latent targets for the future-prediction loss.","marker":"[19]"},{"why":"DINOv2 is one half of the fused ViT visual encoder used by ReflexVLA.","marker":"[16]"},{"why":"SigLIP is the other half of the fused ViT visual encoder used by ReflexVLA.","marker":"[17]"},{"why":"LIBERO provides the static manipulation benchmark used to show ReflexVLA does not sacrifice generalization.","marker":"[12]"},{"why":"A dynamic-manipulation baseline whose 50.2% ReflexBench average is the closest competitor to ReflexVLA's 50.4%.","marker":"[5]"},{"why":"SmolVLA serves as a lightweight baseline and as the probe model for the inference-frequency, chunk-size, and action-horizon studies.","marker":"[8]"},{"why":"OpenVLA-OFT is the 7B generalist baseline that ReflexVLA outperforms on ReflexBench.","marker":"[20]"},{"why":"pi0.5 is the 4B generalist baseline that ReflexVLA outperforms on ReflexBench.","marker":"[21]"},{"why":"DynamicVLA is a dynamic-manipulation baseline motivated by perception-execution delay, which ReflexVLA also outperforms.","marker":"[4]"}],"fun_headline_variants":["Future-state prediction lifts 1B VLA to 50.4% success","Reactive manipulation: compact VLA outdoes larger rivals","ReflexBench exposes latency, small model thrives","Small VLA, big reaction gains: 30% to 50.4%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison stands or falls on whether ReflexBench's injected delays and its planner-and-reinforcement-learning demonstration pipeline reproduce real deployment conditions without systematically favoring ReflexVLA's own predictive and multi-frame design.","fun_headline_variants_meta":{"raw":{"variants":["Future-state prediction lifts 1B VLA to 50.4% success","Reactive manipulation: compact VLA outdoes larger rivals","ReflexBench exposes latency, small model thrives","Small VLA, big reaction gains: 30% to 50.4%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1665,"prompt_tokens":918,"completion_tokens":747,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":671}},"tokens_in":534,"tokens_out":747,"duration_ms":7021,"temperature":1.0,"reasoning_tokens":671,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T21:12:35.538866+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the same policies on a physical conveyor or ball-catching setup with a measured wall-clock action cycle, then check whether ReflexBench success rates under Real-Time-Factor-matched latency rank the policies the same way; a reversed ranking, or a failure of the injected delay to reproduce real timing sensitivity, would falsify the claim that the benchmark faithfully evaluates reaction-critical manipulation.","supporting_citations":[{"cited_title":"Vla-adapter: An effective paradigm for tiny-scale vision-language-action model,","cited_arxiv_id":null,"evidence_quote":"VLA-Adapter is the backbone ReflexVLA extends; its 30.3% ReflexBench average is the direct baseline ReflexVLA raises to 50.4%."},{"cited_title":"Dinov2: Learning robust visual features without supervision,","cited_arxiv_id":null,"evidence_quote":"DINOv2 is one half of the fused ViT visual encoder used by ReflexVLA."},{"cited_title":"π 0.5: a vision-language-action model with open-world generalization,","cited_arxiv_id":null,"evidence_quote":"pi0.5 is the 4B generalist baseline that ReflexVLA outperforms on ReflexBench."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}