{"id":"a51dabca-6b52-47bf-a6ac-dcf1c7a9ac93","arxiv_id":"2611.07964","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Time-indexed skill success models, trained on branched rollouts at multiple launch delays, improve whole-task success under 100-300 ms latency and stay calibrated where static models are overconfident.","lead":"Robots that manipulate moving objects act on information that is already outdated, so a skill that looked good at the moment of observation may fail when it actually starts. This paper trains success models that take the launch delay into account, and a planner that uses them gains 2.8 to 6.2 percentage points over latency-blind baselines in simulation and exposes a severe overconfidence problem in a real-robot test.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Planner gains rest on MuJoCo alone and no artifact is released; the real-robot test intentionally avoids the planner, so the headline comparison is not independently validated.","rationale":"Read in good faith: the paper defines G_k(z, τ) cleanly, branched rollouts provide exact labels under the stated ready-pose idling condition, the planner comparison is paired over the same 2,000 episodes with episode-bootstrap confidence intervals, and replication over five runs is reported transparently. Proposition 1 and the forecast-error expansion in Equation (4) correctly identify when forecast baselines should suffice, and the hardware result in which a matched physics forecast slightly beats the time-indexed model is consistent with that theory rather than a contradiction. The most load-bearing risk is not a mathematical slip but the gap between the central quantitative claim (planner gains) and the evidence: the hardware test deliberately does not exercise the planner, and no artifact is released for independent checking. Because the authors state this limitation explicitly, I do not treat it as a hidden flaw; I also do not see an internal inconsistency that would justify rejection. The reader's CONDITIONAL verdict is appropriate; this concern strengthens the reason for the condition rather than changing it, so I would keep the verdict unchanged.","tokens_in":26049,"tokens_out":28043,"duration_ms":483329,"concrete_test":"Run the paper's planner comparison on the real UR5e for the slow-launch band at ℓ = 200 ms, using the authors' own power estimate of roughly 380 trials per planner (two-sided α = 0.05, 80% power, Section VIII), with the TI-flat versus B1 and TI-flat versus B1s paired differences pre-registered before data collection. If the 95% interval for either difference includes zero, the central quantitative claim is not confirmed outside simulation; if both intervals exclude zero, the concern is settled in the paper's favor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the receding-horizon planner's 2.8 to 6.2 point advantage over static and forecast baselines at 100 to 300 ms of latency (Table I, S4, S6). For this gain to carry the paper's real-robot motivation, the MuJoCo evaluation with analytic skills and a synthetic tracker must be representative. The hardware experiment (Section VII) is explicitly scoped to calibration: one skill, one delay, 49 launches on one day, and the authors state it is not meant to show planner outperformance (b174, b230). It therefore does not exercise the central quantitative claim. The authors acknowledge this limitation, but what remains unresolved is that no code, data, or independent implementation is released, so the headline comparison cannot be checked from the text alone, and no term of the planner comparison has out-of-simulation validation. The fact that the one real-robot comparison actually run (hardware-matched forecast versus time-indexed, Section VII-C) slightly favors the forecast is consistent with the conditions in Proposition 1 and Equation (4), but it also underscores that the relative gain over forecast-based baselines is task-dependent; the only task in which the planner gain is demonstrated is synthetic. This is a scope and reproducibility risk, not an internal inconsistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper addresses the problem of latency-induced state staleness in skill-level manipulation planning. It proposes time-indexed skill success models G_k(z, tau), which predict whole-task success when skill k is launched tau seconds after observing state z, and trains them with rollouts branched at a grid of launch delays while the robot holds a ready pose. These models feed a receding-horizon planner that scores (skill, extra wait) pairs, averaging over the latency distribution when only its distribution is known. Proposition 1 and Eq. (4) are used to characterize when evaluating a static model on a forecast is sufficient and to explain its failure as forecast spread grows. In MuJoCo, with a UR5e capturing a sliding puck using three analytic skills, the best time-indexed planner TI-flat exceeds static, forecast, and rule baselines by 2.8 to 6.2 points at 100 to 300 ms latency, with positive paired bootstrap intervals across five runs. It is calibrated on delayed launches (ECE 0.009 vs 0.239 for the static model), and a frozen-clock evaluation overstates success by up to 23 points. A hardware rolling-ball test with one skill and one delay confirms the predicted calibration gap: a static model predicts 0.99 on the 35 delayed launches it rates at least 0.8, of which 63% succeed, while the time-indexed model predicts 0.56.","tokens_in":26395,"tokens_out":17586,"duration_ms":114616,"significance":"The core idea is simple, clearly formulated, and a natural extension of competence-model planning: instead of asking whether a skill succeeds if launched immediately, ask whether it succeeds if launched after the latency. If the results hold, this gives an actionable way to make skill selection robust to perception and computation delay, and the branched-rollout construction is a clean way to generate exact labels under the stated idling assumption. The evaluation is unusually careful: paired episodes, bootstrap intervals, five complete runs, a frozen-clock control, and a hardware protocol with a pre-specified subset and an a priori success criterion. The Proposition 1/Eq. (4) analysis provides a useful diagnostic for when forecast-based correction is sufficient, and the hardware result is reported honestly, including the regime where a matched forecast is competitive. The main residual risk is that the headline planner gain has no out-of-simulation validation and the implementation is not released, which limits independent checking of the central quantitative claim.","major_comments":[{"comment":"Equation (2) and the surrounding text say that the planner selects a skill k* and an extra wait delta*, with the skill starting at t + ell + delta*. In Algorithm 1, however, a nonzero delta* is never executed: the robot only holds the ready pose and re-decides after max(Delta_t, ell), and a launch is triggered only when delta* = 0, the wait exceeds delta_max, or the object is lost. The implemented policy is therefore a one-step wait-and-replan rule ('launch now if the best candidate delay is zero, otherwise wait one control period and replan'), not a planner that commits to the launch time returned by Eq. (2). Please state this explicitly and define the candidate wait as part of an inner optimization whose first action is executed; otherwise the reader cannot tell which policy generated Table I.","section":"Section IV-C and Algorithm 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The training grid T stops at 1.3 s, while the planner can query G_k for launch delays up to ell + delta = 1.5 s under the fixed 300 ms latency, and up to about 1.7 s under the capped heavy-tailed latency (or at least 1.45 s with the largest mid-quantile used by the -dist planners). Section VIII says the models 'interpolate between nine grid delays', but the optimization in Eq. (2) can evaluate the model outside its training support. Please restrict candidate waits so that ell + delta stays inside T, extend the grid, or state and justify the extrapolation; the calibration and exact-label guarantees are established on the grid, not beyond it.","section":"Section IV-C and Table S1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The central quantitative claim is the planner advantage in Table I and Tables S4/S6, but this comparison is entirely within MuJoCo with analytic skills and a synthetic tracker. The hardware experiment is explicitly scoped to calibration and does not exercise the planner, as the authors acknowledge in Section VII and Section VIII, and I agree that a hardware planner comparison would require an impractical number of trials. What remains unresolved is reproducibility: no code, data, trained models, or simulator configuration is released, so the headline comparison cannot be independently checked from the text. Given the modest 3 to 6 point effect sizes, I ask for a release artifact or at minimum a detailed protocol with exact simulator settings and model checkpoints so that the central claim is verifiable.","section":"Sections VII and VIII","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The instruction to 'decide again after max(Delta_t, ell)' is ambiguous in the random-latency setting, where the planner knows only the distribution of ell but not the realized value. Please specify whether this schedule uses the realized latency, an upper bound, or a quantile, and how it is implemented for the -dist variants.","section":"Algorithm 1, step 10","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The text says TI-flat waited for at least one decision period in 28 to 36% of episodes. Because the loop in Algorithm 1 waits max(Delta_t, ell) rather than Delta_t, it would help to state explicitly how a 'decision period' is counted in this statistic, especially under random latency.","section":"Section VI-B and Table I","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Two things you should know. The core idea is real: a skill success model that takes launch delay τ as an input, G_k(z, τ), trained with a branching rollout recipe, driving a planner that chooses both skill and wait time. The cited competence and delay-compensation literature does not contain this delay-indexed formulation or the branched labeling procedure. And the paper is honest in ways that matter — it says plainly where its evidence stops, and its evidence stops short of the headline claim in one respect.\n\nWhat it does well. Proposition 1 is one line from the tower property, but it earns its place: it shows a static model on a point forecast is biased, equation (4) puts the error on the forecast covariance, and the pair predicts exactly the regime where a matched physics forecast suffices — which the hardware test then shows. The branched rollout scheme is a reusable recipe: M+1 rollouts per episode yield (M+1)(M+2)/2 exact (state, delay, outcome) pairs because the robot idles at the ready pose. The evaluation protocol is careful: paired episodes, bootstrap intervals, five complete runs across seeds and shifted dynamics, and a frozen-clock control showing up to 23 points of overstatement. The hardware result is the most striking single finding: a static model trained in a matched simulator predicted 99% on 35 delayed launches, 63% succeeded, and the time-indexed model predicted 56%. Brier over all 49 launches fell from 0.260 to 0.031.\n\nSoft spots, in proportion. The headline planner gains (2.8–6.2 points over the best baselines at 100–300 ms) come from MuJoCo alone, and no code or data are released to check them. The stress-test note is fair: the one real-robot comparison that runs a forecast against the time-indexed model slightly favors the forecast (Brier difference −0.011, interval excluding zero). The authors explain this via Proposition 1 — small forecast spread relative to a 16 mm tolerance — but it still caps the claim. They flag all of this in Section VIII, including that the hardware test was never meant to exercise the planner. The ready-pose idling assumption is load-bearing for the exactness of the branched pairs; they flag that pre-positioning would break it. The gains are modest in absolute terms, and they say so. Citation pattern looks clean.\n\nI agree with the reader's CONDITIONAL verdict. The rigor is real; the significance is capped by simulation-only planner evidence and missing artifacts. Who it is for: anyone working on skill selection, competence models, or calibration under latency. It deserves a serious referee. My recommendation: send it, and make artifact release plus some out-of-simulation check of the planner the explicit conditions for acceptance.","headline":"Delay-indexed success models are a genuine increment with careful simulation evidence; the planner gains rest on MuJoCo alone, no artifacts are released, and the paper deserves review with those as conditions.","tokens_in":26938,"tokens_out":6727,"duration_ms":42951},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper claims that robot skill success should be modeled as a function of the launch delay, not just the observed state; a planner using such models beat static and forecast-based baselines by 2.8-6.2 points at 100-300 ms latency.","keywords":["time-indexed skill success models","latency-aware skill selection","dynamic manipulation","branched rollouts","receding-horizon planning","calibration under delay","planning under uncertainty","skill launch timing"],"falsifier":"Re-run the simulation planner comparison with the robot allowed to pre-position toward the puck during the latency wait instead of idling at the ready pose: the branched-rollout pairs then stop being exact samples of $G_k(z, \\tau)$, so the claim predicts the time-indexed planner's 2.8-6.2 point advantage should shrink or vanish. A complementary check is to repeat the hardware protocol with doubled acceleration perturbations; the claim predicts the static model stays severely overconfident on its confident delayed subset while the time-indexed prediction tracks the observed rate, so a static-model mean prediction that falls inside the observed Wilson interval would refute the calibration claim.","tokens_in":27495,"feed_emoji":"🤖","tokens_out":18881,"duration_ms":102790,"temperature":0.7,"pith_summary":"The paper claims that a robot manipulating a moving object should score each skill by its whole-task success when launched after a delay, not by the success of an immediate launch from the observed state, because 100-300 ms of perception-to-action latency can change which skill is best. It defines the time-indexed success model $G_k(z, \\tau)$, the probability that the task succeeds if skill $k$ starts $\\tau$ seconds after observing state $z$, and shows that evaluating a static competence model on a forecast of the launch state is exact only in expectation: a point forecast errs roughly in proportion to the forecast spread times the local curvature of the success function. The enabling mechanism is branched rollouts, which give exact training labels because the robot idles at a ready pose, so a rollout launched from each grid-time snapshot is a valid sample for every earlier decision. A receding-horizon planner using the time-indexed model beat static, forecast-based, and rule-based planners by 2.8-6.2 percentage points at 100-300 ms injected latency in simulation, with all 80 paired 95% intervals above zero across five runs, and was calibrated on delayed launches where the static model was not (ECE 0.009 versus 0.239). On a real robot catching a rolling ball, the static model predicted 99% success on its 35 confident delayed launches, of which only 63% succeeded, while the time-indexed model predicted 56%.","feed_headline":"Launch-delay indexing beats static planners by up to 6 points","feed_subtitle":"Under 100-300 ms latency the delay-indexed planner added 2.8-6.2 points and stayed calibrated while static models overpredicted.","key_machinery":"The central object is the time-indexed skill success model $G_k(z, \\tau)$ — the probability that the whole task succeeds if skill $k$ is launched $\\tau$ seconds after the decision state $z$ was observed — together with the identity $G_k(z, \\tau) = \\mathbb{E}[G_k(z_{t+\\tau}, 0) \\mid z_t = z]$, which holds because the robot idles until launch and the decision state is a sufficient statistic. This identity turns the delay problem into a function-approximation problem: the delay enters as an input to a classifier, and the model needs no forecast model at run time. The training mechanism that supplies the exact labels is branching: the simulator runs once to each grid delay, a snapshot is stored, every skill is launched from every snapshot, and each rollout's outcome labels all earlier decisions, producing $(M+1)(M+2)/2$ exact pairs per episode (45 for the nine-delay grid). Around this sits a receding-horizon planner that every 0.1 s chooses the skill and extra wait maximizing the expectation of $G_k$ under the latency distribution, using eight equal-probability quantiles when only the distribution is known.","core_discovery":"The central claim is that latency-aware skill selection should use the whole-task success probability as a function of the decision state and the launch delay, $G_k(z, \\tau) = \\Pr(\\text{success} \\mid z_t = z,\\ k \\text{ starts at } t+\\tau)$, and that this function is learnable exactly from branched rollouts. Proposition 1 states that when the robot idles until launch and $z_t$ is a sufficient statistic, $G_k(z, \\tau) = \\mathbb{E}[G_k(z_{t+\\tau}, 0) \\mid z_t = z]$, so forecast-based correction is exact only on average: to second order the point-forecast error is $\\tfrac{1}{2}\\mathrm{tr}(\\nabla_z^2 G_k(\\hat z_\\tau, 0)\\, \\Sigma_\\tau)$, large where success changes sharply, for example at the edge of a capture tolerance. The training mechanism is branching: with the robot held at its ready pose the world evolves identically whether or not a skill will launch, so a rollout branched at grid delay $\\tau_j$ yields an exact labelled pair $(z_i, \\tau_j - \\tau_i, y)$ for every earlier decision, giving $(M+1)(M+2)/2$ pairs per episode with no off-policy correction. The resulting TI-flat planner was the best planner at every tested latency, exceeding the static model by 2.9-5.3 points, the sampled-forecast model by 2.8-5.7, the point-forecast model by 4.0-6.2, and the rule-based planner by 3.9-7.2 at 100-300 ms, with every paired 95% interval above zero in all five runs, and a factored variant differing from the static baseline only by the delay input gained 1.3-4.9 points. The claim also includes a prediction about calibration: on delayed launches the time-indexed model was calibrated (ECE 0.009) where the static model was severely overconfident (ECE 0.239), and the hardware test reproduced the gap, with the static model's 99% mean prediction on its confident delayed subset corresponding to a 63% observed success rate.","pith_inferences":["The identity (3) suggests a quantitative criterion for when time-indexing matters most: the gain over forecasting should grow with the ratio of forecast spread to the task's success-tolerance curvature, which is why the hardware demo (millimetre spread versus a 16 mm tolerance, one skill) showed near-parity while the three-skill simulation showed the largest gaps.","The branched-rollout trick transfers to any task where the robot can commit to idling while the world evolves, but as soon as the delay is used productively — pre-positioning, or a learned policy acting before launch — exact labels disappear and the method would need importance weights or a pre-launch policy model; the authors flag the break but not the repair.","A cheap diagnostic follows from the hardware result: systems that output competence scores for dynamic tasks should be audited by binning predicted success against observed outcomes conditioned on inference delay, since even a model that is calibrated at zero delay can be badly overconfident on delayed launches.","The same delay-indexing could be folded into the confidence heads of vision-language-action policies, making competence estimates a function of expected start time rather than a post hoc forecast correction."],"forward_implications":["Any static competence model can be upgraded to a time-indexed one without new online machinery: the same observation is fed to the network with $\\tau$ appended, and the training data come from branched rollouts taken while the robot idles.","Forecast-based delay compensation is sufficient only when the forecast spread is small relative to the skill's tolerance; the second-order error formula identifies where point forecasting breaks, and the experiments show sampled forecasts recover only part of the loss.","Evaluation protocol matters: freezing the world during latency overstated adaptive planners by 6 to 23 points and hid most of the reported differences, so dynamic-manipulation benchmarks should keep the clock running during inference.","Knowing only the distribution of the per-decision latency captured most of the attainable benefit, since telling the planner the realized latency of each decision added at most about one point.","A single temperature rescale cannot repair static models across delays, because recalibrating for delayed launches degraded zero-delay calibration; the overconfidence is structural, not a scaling artefact."],"supporting_citations":[{"why":"Supplies the 100-300 ms end-to-end inference latency that defines the operating regime the paper targets.","marker":"[1]"},{"why":"Represents the skill-competence-model tradition (initiation networks and symbols) that the paper extends by adding the launch delay as an input.","marker":"[4]"},{"why":"The Smith predictor, ancestor of predict-then-act delay compensation, of which the B1p and B1s baselines are skill-level instantiations.","marker":"[9]"},{"why":"Future-state-conditioned asynchronous policy, the continuous-policy analogue against which the skill-level forecast baseline B1p is compared.","marker":"[10]"},{"why":"Counterfactual world-action planning for dynamic manipulation, cited with [9], [10] as the forecast-evaluation lineage behind B1p.","marker":"[25]"},{"why":"Supplies temperature scaling and the expected-calibration-error machinery used to train and evaluate every classifier.","marker":"[29]"},{"why":"The physics engine in which the main quantitative planner comparison runs.","marker":"[31]"},{"why":"The bootstrap resampling used for every paired success interval and calibration error bar.","marker":"[33]"},{"why":"The Wilson interval used for the hardware observed-success rate on the 35 confident delayed launches.","marker":"[35]"}],"fun_headline_variants":["Delay-aware skill model beats static planners by 2.8-6.2 points under latency","Predict skill success at launch time, not observation time, to plan better under latency","Calibrated launch-delay model beats static planners, avoids overprediction","Time-indexed model picks the launch delay and gains up to 6.2 points","Delayed-launch success model adds 2.8-6.2 points across tested latencies"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise is that the robot does nothing during the latency and the chosen wait, so a rollout branched from a later snapshot is an exact training label (and the identity $G_k(z, \\tau) = \\mathbb{E}[G_k(z_{t+\\tau}, 0) \\mid z_t = z]$ holds) for every earlier decision; if the robot pre-positions, moves during the delay, or runs a learned policy before launch, that exactness breaks and the method must be reworked, as the authors acknowledge.","fun_headline_variants_meta":{"raw":{"variants":["Delay-aware skill model beats static planners by 2.8-6.2 points under latency","Predict skill success at launch time, not observation time, to plan better under latency","Calibrated launch-delay model beats static planners, avoids overprediction","Time-indexed model picks the launch delay and gains up to 6.2 points","Delayed-launch success model adds 2.8-6.2 points across tested latencies"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.001028,"raw_usage":{"cost":0.0010281908,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0010281908,"upstream_inference_prompt_cost":3.95668e-05,"upstream_inference_completions_cost":0.000988624},"total_tokens":6862,"prompt_tokens":1430,"completion_tokens":5432,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":1244,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":5332}},"tokens_in":1430,"tokens_out":5432,"cost_source":"openrouter_usage","duration_ms":57599,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":5332,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-06T11:59:12.425926+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Re-run the simulation planner comparison with the robot allowed to pre-position toward the puck during the latency wait instead of idling at the ready pose: the branched-rollout pairs then stop being exact samples of $G_k(z, \\tau)$, so the claim predicts the time-indexed planner's 2.8-6.2 point advantage should shrink or vanish. A complementary check is to repeat the hardware protocol with doubled acceleration perturbations; the claim predicts the static model stays severely overconfident on its confident delayed subset while the time-indexed prediction tracks the observed rate, so a static-model mean prediction that falls inside the observed Wilson interval would refute the calibration claim.","supporting_citations":[{"cited_title":"Real-time execution of action chunking flow policies,","cited_arxiv_id":null,"evidence_quote":"Supplies the 100-300 ms end-to-end inference latency that defines the operating regime the paper targets."},{"cited_title":"From skills to symbols: Learning symbolic representations for abstract high-level planning,","cited_arxiv_id":null,"evidence_quote":"Represents the skill-competence-model tradition (initiation networks and symbols) that the paper extends by adding the launch delay as an input."},{"cited_title":"Closer control of loops with dead time,","cited_arxiv_id":null,"evidence_quote":"The Smith predictor, ancestor of predict-then-act delay compensation, of which the B1p and B1s baselines are skill-level instantiations."},{"cited_title":"VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference","cited_arxiv_id":"2512.01031","evidence_quote":"Future-state-conditioned asynchronous policy, the continuous-policy analogue against which the skill-level forecast baseline B1p is compared."},{"cited_title":"Dynamic Manipulation with World-Action Models via Counterfactual Planning","cited_arxiv_id":"2609.33172","evidence_quote":"Counterfactual world-action planning for dynamic manipulation, cited with [9], [10] as the forecast-evaluation lineage behind B1p."},{"cited_title":"On calibration of modern neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies temperature scaling and the expected-calibration-error machinery used to train and evaluate every classifier."},{"cited_title":"MuJoCo: A physics engine for model-based control,","cited_arxiv_id":null,"evidence_quote":"The physics engine in which the main quantitative planner comparison runs."},{"cited_title":"Efron and R","cited_arxiv_id":null,"evidence_quote":"The bootstrap resampling used for every paired success interval and calibration error bar."},{"cited_title":"Probable inference, the law of succession, and statistical inference,","cited_arxiv_id":null,"evidence_quote":"The Wilson interval used for the hardware observed-success rate on the 35 confident delayed launches."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}