{"id":"39fc62ff-af86-41e5-95c6-fdee2a26970e","arxiv_id":"2610.24579","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"LIBERO-Dynamic introduces controllable object motion, calibrated inference latency, and multiple execution strategies to enable reproducible benchmarking of robot policies in dynamic environments.","lead":"LIBERO-Dynamic extends the LIBERO simulation benchmark to include moving objects and realistic inference delays. It provides a controlled platform for evaluating robot policies in dynamic manipulation scenarios.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified; the benchmark is well-constructed and limitations are openly discussed.","rationale":"The reader's weakest assumption (median-calibrated latency) is correctly identified and acknowledged by the authors. This concern is real but not load-bearing for the paper's central claim, because the benchmark is designed as a controlled, reproducible testbed. The paper's experiments are internally consistent, limitations are transparent, and the code and data are released. No hidden contradiction or unsupported inference was found. The reader's ACCEPT verdict with HIGH confidence is appropriate.","tokens_in":19466,"tokens_out":2997,"duration_ms":32211,"concrete_test":"To further validate the platform, one could compare key results (e.g., Table 2 model ranking or Figure 4 Async sweep) when latency is drawn from the empirical distribution (e.g., random sample per inference) instead of fixed median. If the relative model ordering or qualitative findings (e.g., underflow threshold) change significantly, this would quantify the sensitivity and guide future benchmark design.","verdict_should_be":"UNCHANGED","load_bearing_attack":"After careful review, I do not identify a load-bearing concern that undermines the central claim. The paper's construction is methodical, the benchmark dimensions are clearly specified, and the limitations are explicitly stated (b142–b145, b307, b308, b321, b327). The weakest assumption—using median-calibrated latency—is acknowledged by the authors, and the benchmark is designed for reproducibility rather than exhaustive realism. The static-to-dynamic demonstration conversion is hand-designed but internally consistent for the benchmark's goals. Fine-tuning comparisons are appropriately caveated. The paper supports its claim of providing a reproducible platform for evaluating and improving policies in dynamic manipulation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper introduces LIBERO-Dynamic, a benchmark that extends the 40 LIBERO manipulation tasks to dynamic environments where objects move along prescribed trajectories (linear, curvilinear, zigzag) at three speeds (0.03, 0.06, 0.09 m/s). The benchmark accounts for model-specific inference latency by advancing the physics simulation during policy computation. It includes 9,000 quality-filtered dynamic demonstrations generated from static demonstrations via online motion compensation. The paper evaluates six pretrained policies on 1,800 conditions per configuration, compares synchronous, asynchronous, real-time chunking (RTC), and VLASH execution strategies, and fine-tunes three architectures using dynamic and mixed data. A small-scale real-robot study on four tasks provides complementary evidence. The central claim is that LIBERO-Dynamic provides a reproducible platform for evaluating and improving robot policies for moving-target manipulation, and the experiments show that dynamic performance depends on both model choice and timing of inference and execution.","tokens_in":19742,"tokens_out":1785,"duration_ms":20253,"significance":"If the claims hold, LIBERO-Dynamic fills a clear gap in existing manipulation benchmarks by making object motion, inference latency, and execution timing explicit and controllable. This is important because most simulation benchmarks treat policy inference as instantaneous and targets as static, obscuring a critical temporal dimension for deployment. The benchmark's controlled setup (40 tasks, three motion families, three speeds, latency-calibrated protocols) enables systematic comparisons that were previously difficult. The paper is transparent about limitations (oracle horizon selection, median latency, non-isolated data composition in fine-tuning, small-scale real-robot validation), which strengthens trust. The construction of 9,000 dynamic demonstrations and the open-source release of code and configurations further increase the benchmark's utility. The cross-model evaluation, execution strategy analysis, and fine-tuning experiments provide concrete empirical findings (e.g., speed sensitivity differs across models, queue underflow limits async gains, temporal conditioning interacts with dynamic data) that are valuable for the community.","major_comments":[],"minor_comments":[{"comment":"The description of the asynchronous queue blending weights (0.3 old, 0.7 new) and the prefetch trigger ratio (0.5K) would benefit from a brief justification or citation to show that these values are standard or effective. The current text presents them as evaluation settings without motivation.","section":"§3.4, Latency-Aware Execution Protocols","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 2 reports a 'Ret.' column that normalizes dynamic All SR by the speed=0 result. The speed=0 result is obtained from a separate evaluation (presumably the static baseline). Clarify whether the speed=0 evaluation uses the same episodes with zero object motion, and whether that evaluation is subject to the same latency calibration. The caption says 'Static baseline' but the table header says 'speed= 0'; please ensure consistent terminology.","section":"§4.1, Cross-Model Evaluation under Dynamics","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The fine-tuning experiments for π0.5 use different training steps for dynamic-only (1,500) and mixed (3,000) recipes. The paper notes this and states that the comparison is descriptive. However, it would be helpful to include a controlled comparison where both recipes use the same number of steps (e.g., both at 3,000) to better isolate the effect of data composition. This is not a requirement but could strengthen the analysis.","section":"§4.3, Training with Dynamic Data","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"In Figure 4, the zero-latency synchronous curve (panel a) is labeled as 'τ = 0' but it actually represents an idealized diagnostic where inference latency is set to zero. The caption says 'τ = 0', which is clear, but the legend line says 'τ = 0' without further explanation. Consider adding a footnote that this is a diagnostic condition, not a realistic deployment setting.","section":"Figure 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The retry cascade logic is described in detail, but the number of fallback attempts per condition is not reported. A brief summary of how many conditions required fallbacks and how many were ultimately excluded would help readers assess the coverage of the generated dataset.","section":"§A.3, Dynamic-Demonstration Generation Details","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"accept"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The key thing to know: this paper extends LIBERO with controlled object motion and latency-aware execution, producing a reproducible benchmark for dynamic manipulation. It is thorough, honest about its limitations, and should be accepted.\n\nWhat is new: LIBERO-Dynamic adds three motion families and three speed tiers to all 40 LIBERO tasks, with model-specific inference latency calibrated and baked into the simulation loop. It compares six pretrained policies, multiple execution strategies (sync, async, RTC, VLASH), and action horizons. It also generates 9,000 dynamic demonstrations from static demos via online motion compensation and shows that fine-tuning on dynamic data can improve performance, albeit with task- and architecture-dependent gains.\n\nCredit where due: the engineering is solid. The friction-driven contact platform is a plausible way to add motion without breaking the simulator. The latency calibration procedure is documented in detail, and the execution strategy comparisons are controlled (same checkpoint, same episodes). The paper is unusually transparent about its own limitations—it explicitly notes that it does not cover latency variance, that the fine-tuning recipes differ in steps, that best horizons are measured rather than validated, and that real-robot validation is small-scale. This honesty is a strength, not a weakness.\n\nSoft spots: the incremental novelty is real. DOMINO, DynamicVLA/DOM, and ReflexBench already cover dynamic manipulation with latency awareness. LIBERO-Dynamic’s contribution is the scale (40 tasks), the systematic comparison of execution strategies, and the data mixture study. The static-to-dynamic conversion is hand-designed and condition-matched, so the fine-tuning results primarily measure adaptation to known dynamics, not generalization to unseen motion. The real-robot validation covers only four conditions on one platform and is consistent with simulation rather than providing fundamentally new evidence.\n\nNone of these are load-bearing flaws. The paper is careful and the claims are appropriately bounded.\n\nWho benefits: robotics researchers who need a standardized testbed for evaluating policies on moving targets, especially those studying inference latency and execution timing. The benchmark code and data are released.\n\nVerdict: This paper deserves a serious referee and should be accepted. I would support it after minor revisions—perhaps adding a brief discussion of how the results might change if latency variance were modeled, or a clearer separation of the fine-tuning step counts in the main text. But the paper is ready as is.","headline":"LIBERO-Dynamic is a well-engineered benchmark that fills a clear gap; limitations are openly discussed, no red flags, deserves acceptance with minor revisions.","tokens_in":20211,"tokens_out":1064,"duration_ms":12750},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"LIBERO-Dynamic introduces a controlled benchmark for robotic manipulation of moving objects, showing that policy performance depends on execution timing as much as on the model itself.","keywords":["dynamic manipulation","robotic benchmarking","simulation benchmark","inference latency","action chunking","execution strategy","policy evaluation","LIBERO"],"falsifier":"Run the LIBERO-Dynamic benchmark on a policy that uses a reactive online correction head (e.g., A2C2) alongside the standard chunked execution. If the reactive head fully closes the gap between the best static policy and the best dynamic-fine-tuned policy at high speed (0.09 m/s), then the paper's conclusion that dynamic training provides irreducible gains would be undermined; conversely, if the reactive head still benefits substantially from dynamic training, the case for dedicated dynamic architectures is strengthened.","tokens_in":21009,"feed_emoji":"🤖","tokens_out":2899,"duration_ms":29867,"temperature":0.7,"pith_summary":"The paper establishes LIBERO-Dynamic, an extension of all 40 LIBERO manipulation tasks into dynamic settings where target objects move continuously during robot operation and the simulation world advances according to each model's measured inference latency. The benchmark defines three motion families (linear, curvilinear, zigzag) and three speed tiers, and includes four execution protocols (synchronous hold, asynchronous queue, real-time chunking, future-state-aware streaming). The core finding is that dynamic manipulation success depends on both the policy architecture and the timing of inference relative to execution: faster motion degrades every evaluated model, but the degree varies; the same policy can behave very differently under different execution horizons and queue strategies; and dynamic-data fine-tuning improves performance for all architectures tested, though gains depend on temporal conditioning and training budget. The work also provides a dataset of 9,000 quality-filtered dynamic demonstrations and a small-scale real-robot validation that reproduces timing-related effects observed in simulation.","feed_headline":"Benchmark reveals dynamic manipulation hinges on timing","feed_subtitle":"LIBERO-Dynamic shows that how and when a robot acts matters as much as the model it runs, even with fixed weights.","key_machinery":"The key machinery is a unified simulation loop that couples controlled object motion (driven through MuJoCo contact friction by an invisible platform, never by teleportation) with model-specific, pre-calibrated inference latency. The physical world continues to advance during inference; the number of steps the world advances equals the ceiling of τ×f_c (f_c=20 Hz). The same loop supports both evaluation and training: a static-to-dynamic demonstration converter replays verified static trajectories with online target-relative motion compensation, producing 9,000 filtered dynamic trajectories across 40 tasks, 3 trajectory families, 3 speeds, and 5 motion phases. Four execution protocols (Sync, Async, RTC, VLASH) make the timing of observation, inference, and action explicit and separable.","core_discovery":"The central discovery is that controllable, latency-aware dynamic manipulation benchmarks reveal coupled dependencies between object speed, execution strategy, and action horizon that are hidden in static benchmarks. Six pretrained policies evaluated on 1,800 shared dynamic episodes show that raising object speed from 0.03 to 0.09 m/s lowers success rates by 4.7–16.3 percentage points with model-dependent declines. With a fixed checkpoint, switching between synchronous hold, asynchronous queue, and real-time chunking changes the best-measured success rate from 37.8% to 48.4% on the same tasks; asynchronous replanning only benefits from frequent feedback when the action queue is sustained without underflow. Dynamic fine-tuning of three architectures (π0.5, FastWAM-Joint, and a history-augmented Joint variant) yields gains of up to 21.7 percentage points, but the magnitude depends on the form of temporal conditioning, data composition, and training step count.","pith_inferences":["If the benchmark were extended to include runtime latency jitter (currently acknowledged as a limitation), the observed ordering among execution strategies might change; asynchronous methods that are robust to large jitter could pull ahead of synchronous ones beyond the margins seen here.","The trajectory families (linear, curvilinear, zigzag) differ in direction continuity but do not form a monotonic difficulty scale; future work could design trajectory families that explicitly test predictive tracking vs. reactive correction by controlling acceleration discontinuities.","The static-to-dynamic demonstration pipeline, while tuned for LIBERO, offers a general template for upgrading any static manipulation dataset to dynamic conditions, provided the task has a repeatable reference trajectory; this could multiply the utility of existing large static datasets like DROID or Open X-Embodiment for dynamic scenarios.","The real-robot validation covers only four tasks on one platform; its main value is as a sanity check, but the paper's central claims about timing-dependent performance would be substantially strengthened by a larger-scale real-robot replication, especially under latency jitter."],"forward_implications":["Robot policy evaluation that treats inference as instantaneous or keeps objects static risks overestimating real-world performance; future benchmarks should adopt latency-aware, motion-controlled protocols.","The finding that asynchronous action queues require a minimum horizon to avoid underflow provides a concrete design rule for deploying chunked-action policies in dynamic settings: the horizon must exceed the ceiling of (latency × control rate).","Dynamic fine-tuning improves every tested architecture, but gains vary with temporal conditioning; this motivates architecture-level design for explicit temporal encoding (e.g., history frames, future-state prediction) rather than generic static-to-dynamic fine-tuning.","Execution strategy and action horizon alter the behavior of a fixed checkpoint by up to 10+ percentage points, implying that deployment-time tuning of these parameters is at least as important as model selection for dynamic tasks.","The 1,800-condition benchmark and 9,000-demonstration dataset provide a shared testbed for comparing dynamic manipulation methods, enabling controlled ablations of motion speed, latency, and execution protocol that were previously scattered across different task sets."],"supporting_citations":[{"why":"Provides the base 40-task LIBERO benchmark that LIBERO-Dynamic extends with dynamic motion and latency.","marker":"Liu et al., 2023"},{"why":"DOMINO benchmark, the closest concurrent dynamic manipulation benchmark, used for comparison and positioning of LIBERO-Dynamic's controlled dynamics and data mixture analyses.","marker":"Fang et al., 2026"},{"why":"ReflexBench couples six reaction-critical tasks with configurable latency under synchronous and asynchronous execution; used to situate LIBERO-Dynamic's latency-aware and execution-comparison dimensions.","marker":"Chen et al., 2026b"},{"why":"Introduce the π0.5 and π0 policies that serve as two of the six evaluated pretrained models.","marker":"Physical Intelligence et al., 2025; Black et al., 2024"},{"why":"Introduces VLA-JEPA, a latent-world-model policy evaluated in the cross-model comparison.","marker":"Sun et al., 2026"},{"why":"Introduces OpenVLA-OFT, one of the six evaluated policies, and informs fine-tuning methodology.","marker":"Kim et al., 2025"},{"why":"Introduces FastWAM and FastWAM-Joint, the video-model-based policies evaluated in the benchmark and used in fine-tuning experiments.","marker":"Yuan et al., 2026"},{"why":"LeRobot asynchronous queue implementation, which the paper uses as the producer–consumer structure for its Async execution protocol.","marker":"Cadene et al., 2026"},{"why":"Introduces Real-Time Chunking (RTC), one of the four execution protocols compared in the benchmark.","marker":"Black et al., 2025a"},{"why":"Introduces VLASH, a future-state-aware asynchronous inference method used as a system-level reference in execution-strategy comparisons.","marker":"Tang et al., 2025"}],"fun_headline_variants":["Object speed and execution timing reshape robot policy success","Static benchmarks hide how timing matters for robot manipulation","Faster objects slash robot success by up to 16 percentage points","Same policy, different timing: robot success varies by 10 points","Dynamic benchmark shows action horizon changes robot behavior"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The benchmark calibrates each model's inference latency once as a median of many measurements and then freezes it; it does not account for runtime latency variance, and if such variance qualitatively changes execution (e.g., by causing intermittent queue underflow), the quantitative rankings among models and strategies may not generalize to real deployment where latency fluctuates.","fun_headline_variants_meta":{"raw":{"variants":["Object speed and execution timing reshape robot policy success","Static benchmarks hide how timing matters for robot manipulation","Faster objects slash robot success by up to 16 percentage points","Same policy, different timing: robot success varies by 10 points","Dynamic benchmark shows action horizon changes robot behavior"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"cost":0.0001762684,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0001762684,"upstream_inference_prompt_cost":8.940624e-05,"upstream_inference_completions_cost":8.686216e-05},"total_tokens":1462,"prompt_tokens":984,"completion_tokens":478,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":399}},"tokens_in":984,"tokens_out":478,"cost_source":"openrouter_usage","duration_ms":7015,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":399,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-20T19:20:19.250640+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the LIBERO-Dynamic benchmark on a policy that uses a reactive online correction head (e.g., A2C2) alongside the standard chunked execution. If the reactive head fully closes the gap between the best static policy and the best dynamic-fine-tuned policy at high speed (0.09 m/s), then the paper's conclusion that dynamic training provides irreducible gains would be undermined; conversely, if the reactive head still benefits substantially from dynamic training, the case for dedicated dynamic architectures is strengthened.","supporting_citations":[{"cited_title":"LIBERO : Benchmarking knowledge transfer for lifelong robot learning","cited_arxiv_id":null,"evidence_quote":"Provides the base 40-task LIBERO benchmark that LIBERO-Dynamic extends with dynamic motion and latency."},{"cited_title":"VLA-JEPA: Enhancing vision-language-action model with latent world model.arXiv preprint arXiv:2602.10098, 2026","cited_arxiv_id":null,"evidence_quote":"Introduces VLA-JEPA, a latent-world-model policy evaluated in the cross-model comparison."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}