{"id":"887b74f4-364f-4f6d-a343-7e1d9cdc0ce2","arxiv_id":"2607.27890","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SIDO morphs static demonstrations into counterfactual future-pose samples, training a goal-conditioned policy that, paired with a pose predictor, grasps objects whose motion was unseen during training.","lead":"SIDO is a training-time augmentation that lets robot policies trained only on static-object demonstrations grasp objects that move at test time. It could make imitation-learned robots work on conveyor belts or swaying objects without collecting expensive new demonstrations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central mechanism (policy uses pose channel, not pixels) is asserted but never causally verified; a pose-channel ablation would settle it.","rationale":"The reader's weakest assumption is precisely this issue. I agree that it is load-bearing because the central claim of training on static demonstrations to handle moving objects relies on the policy being goal-conditioned by the predicted future pose, which in turn requires that the pose channel is the operative signal. The paper's t-SNE and tracking slope are suggestive but not causal. An ablation test would settle it. This concern does not change the reader's conditional verdict; it reinforces the condition under which the paper should be accepted. Other issues (error bars, EquiDP+Mug, velocity matching) are secondary and do not threaten the mechanism as directly.","tokens_in":18450,"tokens_out":13821,"duration_ms":147619,"concrete_test":"Zero the object-pose channel input to a trained DP+SIDO policy at deployment (holding images and proprioception fixed) on Gantry x-axis dynamic and Stack x-axis; if success rate and tracking slope β_x collapse relative to the intact policy, the pose channel is causally required. If they do not, SIDO’s gains do not depend on the channel and the §4.2 'by construction' claim is contradicted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §4.2 the paper asserts that 'the policy reads the object’s position from the dedicated observation channel, not from the pixels' and uses this to justify keeping images unchanged during counterfactual augmentation. This is a claim about the learned policy’s internal information flow, not a property of the architecture or training loss. Although the training data may pressure the policy to use the pose channel (since multiple counterfactual samples share the same static image but have different δ), the paper provides no post-hoc verification. At deployment the images show the object at its current position while the pose channel holds the predicted future position; if the policy in fact relies on pixel-level object localization, the training distribution (static image + displaced pose) differs from deployment (moving image + future pose) and the reported success cannot be attributed to SIDO’s mechanism. The t-SNE (Fig. 5) and tracking slope (H2) are consistent with pose-channel use but do not establish causality. This is the load-bearing step because the entire augmentation validity and the interpretation of the empirical gains rest on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SIDO (Static In, Dynamic Out), a counterfactual action augmentation that enables a visuomotor policy trained only on static-object demonstrations to manipulate objects that move at test time. The method takes each static demonstration transition, displaces the object in the dedicated pose channel by a vector δ, and morphs the action chunk so that the hand-object relative pose at the replanning horizon Ta and the prediction horizon Tp is preserved relative to the displaced object. Two variants are presented: H-SIDO, a heuristic linear waypoint ramp, and DynaSIDO, which refines the chunk with MPPI under a learned dynamics model. At deployment, an object pose predictor writes the predicted future object position into the pose channel. Evaluation covers three simulated MimicGen tasks under five motion patterns, two real-world platforms (Gantry, Peachtree), two policy classes (Diffusion Policy and Equivariant Diffusion Policy), and three object pose predictors. The paper claims that SIDO improves moving-object success over baselines while preserving static performance, transfers across policy classes and predictors, and works by preserving the hand-object invariant.","tokens_in":18780,"tokens_out":8403,"duration_ms":79931,"significance":"If the claims hold, this is a practically useful result: it separates moving-object manipulation from expensive moving-object teleoperation and lets the policy be trained once and paired at deployment with any object pose forecaster. The paper has several genuine strengths: the method is clearly specified; the comparison against a policy trained on moving-object demonstrations (Appendix B) is a valuable and often-missing baseline; the evaluation includes two real-world platforms and a speed sweep (Appendix A); and the predictor-swap experiment (Table 4) directly tests the claimed plug-and-play property. The central limitations are evidential: the causal mechanism is asserted rather than verified, the augmentation scale is calibrated to the test speed, and the headline success differences are small relative to the 20-rollout sample size with no confidence intervals or multi-seed results. These issues do not invalidate the idea, but they currently leave the strength of the empirical claims unclear.","major_comments":[{"comment":"The sentence 'By construction the policy reads the object’s position from the dedicated observation channel, not from the pixels' is an empirical claim about the learned policy’s information flow, not a property of the architecture or training objective. The counterfactual observations keep images unchanged and edit only the pose channel; this pressures the policy to use the pose channel but does not guarantee it. At deployment, the images show the object at its current position while the pose channel holds the predicted future position. If the policy relies on pixel-level object localization, the training distribution (static image + displaced pose) diverges from deployment (moving image + future pose), and the reported gains cannot be attributed to SIDO’s mechanism. The t-SNE and tracking-slope evidence is consistent with pose-channel use but does not establish causality. Because the v","section":"§4.2, Fig. 5"},{"comment":"The finite-difference predictor is described as stepping 'along the direction of those median differences with magnitude v Tp' (Appendix F), and Appendix I sets v=0.001, Tp=16, giving ||δ||=0.016 simulation units, which matches the 2 cm/s test speed over 0.8 s (1.6 cm). Thus the predictor’s forecast magnitude and the augmentation scale are both calibrated to the same test speed. The main experiments therefore demonstrate generalization over motion direction, not over speed or displacement magnitude. The abstract’s claim of adapting to 'unseen object motion' overstates this. Appendix C confirms that handling varying velocity requires training over a range of displacement magnitudes and using a learned predictor (FERNN), so this is a real limitation. The paper should either evaluate at speeds outside the augmentation range or explicitly qualify the claim as direction generalization at a kn","section":"Appendix F, Appendix I"},{"comment":"Every success cell in the evaluation is based on 20 rollouts, with no error bars, confidence intervals, or multi-seed variance. Several headline differences are within binomial sampling noise: Peachtree dynamic is 0.60 for DP+SIDO versus 0.45 for DP (cur. pose)—12 versus 9 successes out of 20; Gantry random dynamic is 0.25 versus 0.20; and in Table 4 the FERNN predictor’s 0.40 versus 0.25 for the MLP is 8 versus 5 successes. Additionally, Table 1 shows EquiDP+SIDO is worse than EquiDP (cur. pose) on Mug x-axis and random (0.10 vs 0.35 and 0.00 vs 0.15), which the text attributes to a 'gap between the commanded and executed trajectories' without supporting evidence. These issues directly weaken H1 and H3. Reporting binomial confidence intervals, running multiple seeds, and providing per-condition significance tests would substantially strengthen the empirical claims.","section":"Tables 1, 3, 4"}],"minor_comments":[{"comment":"The conversion from simulation units to physical centimeters is not stated. The reader cannot verify that ||δ||=0.016 simulation units corresponds to 1.6 cm at 2 cm/s; please add the conversion or report δ in physical units.","section":"Appendix I"},{"comment":"There are duplicate table numbers: Appendix A contains a Table 6 (speed sweep) and Appendix B also has a Table 6 (collection cost) and Table 7 (results). Renumber the appendices' tables sequentially.","section":"Appendix A/B"},{"comment":"The tracking slope β_x is reported as a point estimate from a least-squares fit, but no standard error or R² is given. Adding these would make the comparison between SIDO (0.42) and the baselines more interpretable.","section":"§5, H2"},{"comment":"The phrase 'By construction the policy reads...' should be reworded, since it is a design assumption rather than an architectural guarantee. This also aligns with major comment 1.","section":"§4.2"},{"comment":"The statement that DP+AC 'falls to 0.00, below even plain DP' is interesting but the mechanism ('the controller smooths it away') is not demonstrated. A plot of the executed versus commanded trajectory for DP+AC would help.","section":"§5"},{"comment":"The closing remark about an 'approximate Galilean equivariance effect' is vague and not formalized. Either define the symmetry precisely or remove the remark.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a useful and clearly described technique, with a stronger-than-usual evaluation (two real platforms, multiple predictors, moving-object-demonstration baseline). The main risks are the unverified pose-channel mechanism and the speed-calibrated augmentation scale, both of which affect the interpretation of the central claim. I would encourage the editor to require the pose-channel ablation, confidence intervals or seeds, and a qualification of the 'unseen motion' claim before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time. The core idea is simple and probably right: instead of collecting expensive moving-object demonstrations, you take static demos, virtually displace the object, and morph the action chunk so the hand-object relative pose is preserved at the replanning point and the chunk terminus. That turns a dynamic reaching problem into a goal-conditioned one. Clean factorization, and the plug-and-play predictor swap is a nice touch. This is genuinely different from CCIL's learned-dynamics approach and from equivariant spatial priors; it's an explicit geometric-temporal counterfactual. The paper is also honest about its limitations, which I respect.\n\nThe empirical work is broad: three simulated tasks, five motion patterns, two policy classes, two real platforms, a moving-demonstration baseline, and a speed sweep. The gains over baseline are consistent across tasks, with the Stack x-axis jump from 0.15 to 0.90 being the highlight. Static performance is mostly preserved. That's a solid package.\n\nBut there are soft spots, in proportion to how soft they are. First, no error bars anywhere. Every cell is 20 rollouts, and at that size a difference of 0.1 is within binomial noise. The pattern across tasks is convincing, but a referee should ask for confidence intervals or multi-seed runs. Second, the load-bearing mechanism is asserted, not verified. The paper says \"by construction\" the policy uses the pose channel rather than pixels, but that's not a property of the architecture—it's a claim about learned information flow. The training data pressures the policy toward the pose channel, and the t-SNE is consistent with that, but neither proves it. A simple ablation—corrupt or zero the pose channel at test time, or train without images—would settle it. As written, the interpretation of the gains is conditional. Third, the displacement scale v is matched to the test speed: in simulation, v and Tp produce a displacement equal to the 2 cm/s motion over the chunk horizon. That makes the \"unseen motion\" claim less clean. Appendix C partially addresses this by sampling delta over a range, but the main tables all use the matched value. Finally, EquiDP+SIDO on Mug is a real failure—it underperforms EquiDP with current pose on the x-axis and nearly everywhere else—so the \"policy-agnostic\" claim is overstated.\n\nNone of this sinks the paper. The central idea is likely correct, and the weaknesses are addressable. This is a paper for people who care about imitation learning for manipulation in practical settings, and it deserves a serious referee. I'd send it out, but the reviewers should require the pose-channel ablation and some error bars before accepting the strong version of the claim.","headline":"SIDO's core augmentation is new and mostly works, but the paper asserts its mechanism instead of proving it and the 20-rollout tables need error bars before the claim is fully sold.","tokens_in":19213,"tokens_out":3167,"would_cite":true,"duration_ms":34100,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A policy trained only on static object demonstrations can manipulate moving objects by predicting the future pose and preserving the hand-object relative pose, according to SIDO.","keywords":["imitation learning","visuomotor policy","counterfactual data augmentation","moving object manipulation","hand-object relative pose","object pose prediction","diffusion policy","behavior cloning"],"falsifier":"Feed a SIDO-trained policy a static image pair but a pose channel displaced to a fake future position; if the gripper follows the fake position, the pose channel is what matters. Then feed the true moving images but hold the pose channel at the current pose; if the gripper lags or misses, the pixel path is not driving the reach. Either outcome would settle the load-bearing assumption.","tokens_in":18386,"feed_emoji":"🤖","tokens_out":3978,"duration_ms":36592,"temperature":0.7,"pith_summary":"Static In, Dynamic Out (SIDO) claims that a robot policy trained exclusively on demonstrations of stationary objects can manipulate objects that move in ways never seen during training. The key move is to split the problem into predicting where the object will be one action-chunk ahead and learning a goal-conditioned reach that preserves the demonstrated hand-object relative pose. SIDO augments each static demonstration by virtually displacing the object and morphing the action chunk so that the relative pose at the replanning point and chunk terminus is unchanged. At deployment, any object pose predictor supplies the future position, and the policy reaches for it. Experiments across three simulated tasks, five motion patterns, and two real-world platforms report improved moving-object success while keeping static-object performance intact.","feed_headline":"Robots trained on still objects can grab moving ones","feed_subtitle":"SIDO rewrites static demos to hold hand-object pose, then reaches a predicted future spot: no moving-object demos needed.","key_machinery":"The central object is the hand-object relative pose r_t(k) = p_eef_{t+k} - P_{t+k}, defined for offsets k up to the prediction horizon Tp. The invariant constraint preserves r_t(Ta) and r_t(Tp), where Ta is the replanning point and Tp the chunk terminus, when the object is virtually displaced by δ. The counterfactual augmentation edits only the dedicated object-pose observation channel, leaving images unchanged, and morphs the action chunk via a linear ramp (H-SIDO) or an MPPI refinement under a learned dynamics model (DynaSIDO). At deployment, a plug-and-play object pose predictor fills that same channel with the forecast future position.","core_discovery":"The paper's central claim is that the temporal gap between static demonstrations and dynamic deployment can be closed by data augmentation, not by collecting dynamic demonstrations. The hand-object relative pose r_t(k) = p_eef_{t+k} - P_{t+k} is the invariant that carries the argument. For each static transition, SIDO samples a displacement δ, writes the counterfactual future object position into the dedicated pose channel, and morphs the demonstrated action chunk so that r_t(Ta) and r_t(Tp) match the expert's relative pose. The resulting goal-conditioned policy, paired with any object pose predictor, tracks a moving object at rollout. In simulation and on two real-world platforms, SIDO impr","pith_inferences":["If the pose-channel reading holds, the same counterfactual trick could be applied to other sparse, dedicated observation channels (e.g., tactile or proprioceptive features) without retraining the full policy.","The approximate Galilean equivariance the paper observes hints that training over a range of displacement magnitudes may yield policies that generalize to unseen velocities, not just unseen directions — a testable extension beyond the reported speeds.","The collision-ignoring limitation suggests a natural next step: replace the linear or MPPI morph with a collision-aware planner so the displaced reach can navigate clutter, extending SIDO beyond open tabletop scenes.","A causal ablation that zeroes or corrupts the pose channel while leaving images intact would directly test the assumption that the policy reads the pose channel; the paper's t-SNE analysis is correlational, not causal."],"forward_implications":["Manipulation of moving objects no longer requires expensive, unsafe teleoperation on moving targets; large static demonstration datasets become reusable.","A single trained policy can be deployed across different motion regimes by swapping the object pose predictor, with no retraining.","The invariant constraint at replanning and chunk-terminus horizons prevents the compounding out-of-distribution drift that otherwise breaks behavior cloning on moving objects.","Improving the pose predictor directly improves task success, as forecast error and success rate correlate across the tested predictors.","The method is policy-agnostic, holding for both diffusion and equivariant diffusion policies, suggesting broad applicability."],"fun_headline_variants":["Static demos, dynamic reach: SIDO rewrites motion for robots","Teach robots to catch moving things using only static demos","SIDO: counterfactual action augmentation for moving-object grip","Zero new demos: robots learn to track moving objects via SIDO","How to get robots to handle moving objects from static training"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The policy must use the dedicated object-pose channel to decide where to reach, not the pixels; otherwise the training-time mismatch between unchanged images and displaced pose signals diverges from deployment, where images and pose channel both point to the future.","fun_headline_variants_meta":{"raw":{"variants":["Static demos, dynamic reach: SIDO rewrites motion for robots","Teach robots to catch moving things using only static demos","SIDO: counterfactual action augmentation for moving-object grip","Zero new demos: robots learn to track moving objects via SIDO","How to get robots to handle moving objects from static training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1124,"prompt_tokens":726,"completion_tokens":398,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":307}},"tokens_in":470,"tokens_out":398,"duration_ms":4475,"temperature":1.0,"reasoning_tokens":307,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:31:12.992661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed a SIDO-trained policy a static image pair but a pose channel displaced to a fake future position; if the gripper follows the fake position, the pose channel is what matters. Then feed the true moving images but hold the pose channel at the current pose; if the gripper lags or misses, the pixel path is not driving the reach. Either outcome would settle the load-bearing assumption.","supporting_citations":[],"review_version":1}