{"id":"ef589eda-d6e7-4eeb-ae07-5b5afebb2bd9","arxiv_id":"2506.14198","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A three-stage pipeline that turns keypoint tracks into discrete motion tokens, predicts them from action-free video, and decodes them into actions yields large few-shot and zero-shot policy improvements in robot manipulation.","lead":"AMPLIFY is a robot-learning framework that turns ordinary videos into compact motion tokens and uses them to predict how a scene will move, then decodes that motion into robot actions. It aims to let robots learn useful skills from action-free video, reducing the need for expensive, labeled demonstrations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed cross-embodiment benefit from human videos is not isolated: Table 4 compares AMPLIFY trained on robot+human videos against Diffusion Policy trained on robot demos only, so the 1.4x gain could come from the motion-token interface or added robot-video data, not from human transfer.","rationale":"I considered three candidate load-bearing concerns: (a) information loss in the 2D-track/FSQ token interface, which the reader identified as the weakest assumption; (b) the absence of an ablation isolating the human-video contribution in the cross-embodiment experiment; and (c) missing error bars and evaluation-split statements. The information-preservation concern is real but partially addressed by the ATM comparison (AMPLIFY outperforms a pixel-space track policy in few-shot LIBERO) and by the paper's ablation on tokenizer design; it is also explicitly acknowledged in the Limitations section. The error-bar concern affects confidence but not the direction of the main effects. The missing human-video control, by contrast, directly undermines a headline quantitative claim: the 1.4x average real-world improvement is the only direct evidence that action-free human videos help, and the experimental design confounds the human-video addition with the use of the motion-token interface and the availability of robot videos to the forward model. Because this claim appears in the abstract and is highlighted in the contributions, it is the single most load-bearing gap. The proposed check is conservative: it keeps the motion-token interface fixed and varies only the presence of human videos in the forward model's training set. A decisive outcome in either direction is valuable. Since the paper is a preprint without released code or checkpoints, the experiment cannot currently be reproduced from the manuscript, which reinforces the need for this control or for public release of training artifacts. For these reasons I maintain the reader's CONDITIONAL verdict rather than moving it, as the missing control is fixable and does not invalidate the other results (few-shot and zero-action-data generalization) that are less directly affected by this confound.","tokens_in":25895,"tokens_out":6500,"duration_ms":75642,"concrete_test":"Run the real-world protocol underlying Table 4 with three conditions on the same three tasks and demo counts: (1) Diffusion Policy trained on robot demos only, as in the original baseline; (2) AMPLIFY with the Diffusion Policy head, where the forward dynamics model is trained only on the same robot demonstration videos (no human videos) and the inverse dynamics model on the same robot action data; (3) the reported AMPLIFY setup, with forward dynamics trained on robot plus human videos. Keep all other hyperparameters, seeds, and evaluation rollouts (10 per task) identical. If condition 2 approximately matches condition 3, the 1.4x average improvement is not attributable to human videos; if condition 2 approximately matches condition 1 and condition 3 exceeds both, the cross-embodiment claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 3.2 claim a 1.4x average real-world improvement from 'learning from action-free human videos,' and the Cross-Embodiment Transfer paragraph states that training the forward dynamics model on both human and robot video data lets AMPLIFY 'effectively leverage additional human data to learn common dynamics between human and robot motions.' But Table 4 does not include an AMPLIFY variant trained without human videos. The comparison against Diffusion Policy differs in two ways at once: (1) AMPLIFY uses latent motion tokens as an additional conditioning signal, and (2) its forward dynamics model is also trained on the same robot demonstration videos, whereas Diffusion Policy is trained on action labels only. The paper's own video-scaling experiment (Table 14, Appendix F.1) shows that adding robot videos alone improves policy success from 0.12 to 0.55 in a low-data LIBERO setting, so video data alone can produce large gains. Therefore the reported 1.4x average improvement cannot be attributed specifically to cross-embodiment transfer from human videos. This is load-bearing because cross-embodiment learning from action-free human video is one of the paper's headline contributions; if the improvement persists without human videos, the 'learning from human videos' claim is unsupported, and if it disappears, the claim is supported. The reader's rationale explicitly listed this missing control, though it was not the reader's formal weakest_assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AMPLIFY, a three-stage framework for robot learning from action-free videos. It first tokenizes CoTracker keypoint trajectories into discrete FSQ codes via an autoencoder, then trains an autoregressive forward dynamics model to predict future motion tokens from an image and task description, and finally trains an inverse dynamics model to map predicted tokens to actions. Experiments evaluate track prediction accuracy on LIBERO, BridgeData v2, and Something-Something v2; downstream policy learning in in-distribution, few-shot, cross-embodiment, and zero-shot action-label settings; and conditional video prediction. The reported results include a 3.7x reduction in track prediction MSE over ATM, 1.2-2.2x improvements in few-shot policy learning, a 1.4x average real-world improvement when human videos are added, and 60.5% average success on LIBERO target tasks with zero target-task action data.","tokens_in":26189,"tokens_out":7895,"duration_ms":72241,"significance":"If the results hold, AMPLIFY makes a strong contribution by demonstrating that a compact latent motion representation learned from arbitrary videos can provide a useful prior for downstream control, enabling substantial gains in low-data regimes and a novel zero-shot action-label generalization result. The paper is well-structured, includes extensive ablations, and compares against strong baselines. The few-shot and zero-shot experiments include control variants (inverse-only and w/o-tracks) that isolate the motion-token contribution, and the video-scaling experiment in Appendix F.1 is a valuable diagnostic. However, the headline cross-embodiment claim is not supported by the presented comparison, and the track-prediction evaluation split is missing.","major_comments":[{"comment":"The cross-embodiment transfer claim is not isolated: the 1.4x average improvement compares an AMPLIFY policy whose forward dynamics is trained on human and robot videos against a Diffusion Policy trained only on robot demonstrations. This differs in two ways at once (motion-token conditioning and additional video data), so the gain cannot be attributed to human videos. The paper's own video-scaling experiment (Table 14) shows that adding robot videos alone raises success from 0.12 to 0.55 in a low-data LIBERO setting, demonstrating that extra video data alone can produce large gains. To support the claim that action-free human videos provide a cross-embodiment benefit, the authors should add an AMPLIFY variant whose forward dynamics is trained only on robot videos and show that the addition of human videos yields a further improvement over that control.","section":"§3.2, Table 4; Appendix F.1"},{"comment":"The track-prediction evaluation split is not stated anywhere in the paper. It is unclear whether the forward dynamics model is evaluated on videos that were also used for training; if it is, the reported 3.7x MSE improvement and 2.5x pixel-accuracy improvement over ATM are not evidence of generalization. The paper must specify how the videos in each dataset (LIBERO, BridgeData v2, Something-Something v2) were partitioned into training and evaluation sets for the motion tokenizer and forward dynamics model, and confirm that the evaluation rollouts are disjoint from the training rollouts.","section":"§3.1, Table 2; Appendix D"}],"minor_comments":[{"comment":"The claim of 'the first latent keypoint dynamics model' should be qualified with respect to prior work such as Moto [34] and Latent Action Pretraining [33], which also learn latent motion or action representations from videos; the novelty should be positioned more precisely.","section":"Section 1, Contribution 1"},{"comment":"The sentence 'The average improvements of 1.32×, 1.4×, and 1.5×' is confusing because the table reports a single average of 0.42 vs 0.58; the per-task or per-demonstration-count ratios should be reported explicitly.","section":"Section 3.2, Table 4"},{"comment":"The text says 'we evaluate AMPLIFY in both the few-shot setting and the full demonstration setting,' but the main text does not define the exact demonstration counts for 'All' for each task; these are only listed in Appendix Table 9, so a cross-reference would help.","section":"Section 3.2, Cross-Embodiment Transfer"},{"comment":"Success rates are reported without confidence intervals despite being averaged over 10 rollouts; standard errors or per-seed results would make the comparison more rigorous.","section":"Section 3.2, real-world results"},{"comment":"The preprocessing paragraph introduces the window subscript without defining it; e.g., 'length-τ video' and 'τ length-T windows' should be defined precisely.","section":"Appendix D.4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is in scope and the core framework is interesting, but the cross-embodiment claim requires an ablation. Also, please note that related work Moto [34] may undermine the 'first latent keypoint dynamics model' novelty claim; the authors should be asked to differentiate. Overall, I believe a major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBottom line: this is a real contribution and deserves referee time. The architecture—FSQ-quantized keypoint velocities, a video-only forward dynamics transformer, and a separate inverse action decoder—is a sensible and reasonably novel combination. The few-shot LIBERO experiments are the cleanest part: AMPLIFY is compared against ATM and an inverse-only control, and the motion tokens give a consistent edge at 2, 5, and 10 demos (about 1.9x over ATM at 2 demos). The zero-shot result on LIBERO target suites (60.5% average with zero target-task action data) is striking. If that reproduces, it is a meaningful step for learning control from action-free video.\n\nThe main soft spot is the cross-embodiment claim. The abstract and Section 3.2 say training the forward model on human plus robot videos gives a 1.4x real-world improvement, but Table 4 compares AMPLIFY (with human videos) against Diffusion Policy (robot demos only). That differs in two ways at once: the motion-token conditioning and the extra video data. The appendix's own video-scaling experiment shows robot videos alone can move success from 0.12 to 0.55 in a low-data LIBERO setting, so video data alone is a plausible confound. Without an AMPLIFY-without-human-videos variant, the 'learning from human videos' claim is not actually isolated. The fix is easy and should be required.\n\nOther issues are smaller but real: track-prediction tables don't state the evaluation split, main tables have no error bars (and LIBERO-90 gets one rollout per task), and no code or checkpoints are released for a system whose value depends on details. The limitations section candidly flags the 2D-tracks ambiguity and deterministic-dynamics assumption, which is to their credit.\n\nWho it's for: anyone working on video pre-training for manipulation. It is not a ground-up conceptual revolution—each component has antecedents—but the packaging is new and the empirical story is mostly well-controlled. I would accept it for peer review, with the missing human-video control, variance reporting, and code release as conditions. I would not desk-reject it.","headline":"A well-built latent motion token framework with a genuinely interesting zero-shot LIBERO result, but the headline human-video transfer claim needs a missing control before it can be taken at face value.","tokens_in":26772,"tokens_out":2456,"would_cite":true,"duration_ms":25255,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Conditioning a policy on latent motion tokens predicted from video-only training yields 1.2-2.2x low-data gains and the first zero-action-data LIBERO generalization, this paper argues.","keywords":["behavior cloning","action-free video learning","keypoint trajectory prediction","latent motion tokens","forward and inverse dynamics","robot manipulation","cross-embodiment learning","world models"],"falsifier":"Construct a pair of tasks whose keypoint motion is nearly identical over the 16-frame prediction horizon but whose correct actions diverge (for example, two LIBERO-Goal tasks that differ only in which of two objects is the target, filmed so the first 0.8 seconds of motion look alike). If AMPLIFY still solves both at a rate near its reported 60% average, the motion-token interface is more informative than the 2D-ambiguity caveat suggests; if success collapses toward the near-zero inverse-only baseline, the information bottleneck sits in the 2D track representation rather than in the action head.","tokens_in":25656,"feed_emoji":"🤖","tokens_out":13130,"duration_ms":119226,"temperature":0.7,"pith_summary":"AMPLIFY sets out to show that the information needed to control a robot can be carried by action-free video, provided the video is distilled into compact discrete motion tokens. The paper splits policy learning into a forward dynamics model that any videos can train, predicting latent keypoint-motion tokens, and an inverse dynamics model that a small set of action-labeled interactions can train, decoding those tokens into action chunks. On this interface the paper reports 1.2-2.2x improvements in low-data policy learning, a 1.4x average real-world gain from adding human videos over the same action data, and an average 60.5% success rate on LIBERO tasks for which no target-task action data was ever seen. A sympathetic reading is that action generalization is limited less by the volume of action labels and more by the quality of the action-free motion prior.","feed_headline":"Video-only motion priors give robots a 60% zero-shot success rate","feed_subtitle":"Predicting motion tokens from video lifts few-shot learning 1.2-2.2x and unlocks zero-action-data LIBERO tasks.","key_machinery":"The load-bearing object is the latent motion token: single-step velocities extracted from CoTracker tracks of a re-initialized 20 by 20 grid of points, encoded by a causally-masked transformer, quantized with Finite Scalar Quantization (FSQ) into a fixed 2048-code space, and decoded through a local-window classification head that scores each point's next motion inside a 15 by 15 pixel window around its previous location. An autoregressive transformer predicts these tokens from the current image and a language task description, and a separate cross-attention transformer decoder, the inverse dynamics model, maps image, proprioception, and predicted tokens into a distribution over a 16-step action chunk, with temporal ensembling at inference. The discrete codebook and the local-window classification turn motion prediction into a tractable classification problem; the frozen token interface is what allows the forward and inverse stages to train on entirely different datasets.","core_discovery":"The central claim is that latent motion tokens are a sufficient interface between visual dynamics and control: single-step velocities of a re-initialized 20 by 20 keypoint grid are compressed into a discrete 2048-code space, a forward model trained only on videos and task descriptions predicts the next token sequence, and an inverse model that never sees the goal decodes those tokens into an action chunk. The paper argues this decomposition cleanly separates what motion defines a task from how a robot can perform it, letting video data and interaction data scale independently. Empirically the learned dynamics are more accurate than prior keypoint and full-video-prediction baselines, and the gains concentrate exactly where action labels are scarce: few-shot learning, cross-embodiment transfer from human video, and zero-shot generalization on LIBERO suites for which no action data was ever provided.","pith_inferences":["A consequence the paper motivates but does not test: if the motion token is the whole task channel, the inverse model should train on undirected play or exploration data as well as on expert demos, since it never sees goals.","The 2D ambiguity the paper concedes predicts where the method should fail — tasks whose distinguishing information sits in the goal rather than in pixel motion; the LIBERO-Goal set being the hardest zero-shot target (41%) is consistent with that, and swapping in 3D point tracking would be the direct stress test.","The interface framing suggests motion tokens could serve as a reference channel inside other policy families, since the ablations show Gaussian, diffusion, and flow-matching action heads perform nearly identically.","The non-monotonic video-scaling curve (0.34 at 5 videos vs 0.23 at 10) hints that the forward model's benefit depends on track diversity, not raw clip count; a scaling study over internet-scale, scene-diverse video would show whether the cross-embodiment gain saturates."],"forward_implications":["Few-shot learning scales with video rather than action count: with only 2 demonstrations per task, AMPLIFY reaches roughly 1.94x the success rate of the keypoint baseline ATM on LIBERO.","Action-free human video transfers to robot control: adding human demonstrations to the forward model improves real-world success by 1.32x to 1.5x over the same policy fed only robot data.","Zero-action-data generalization is possible: trained on LIBERO-90 actions only, AMPLIFY averages 60.5% success on four unseen LIBERO suites while behavior-cloning baselines score near zero.","Latent motion tokens function as a world-model artifact: conditioning the AVDC video predictor on them improves PSNR, LPIPS, and SSIM on BridgeData v2.","More video, same actions: with action data held at 2 trajectories, task success rises from 0.12 with 2 training videos to 0.55 with 50, evidence that the prior keeps improving as video scales."],"supporting_citations":[{"why":"Supplies CoTracker, the off-the-shelf point tracker that produces the keypoint tracks every later stage consumes as ground truth.","marker":"[36]"},{"why":"Provides Finite Scalar Quantization, the implicit-codebook discretization that defines the 2048-code latent space of motion tokens.","marker":"[58]"},{"why":"ATM is the keypoint-dynamics method whose MSE and pixel accuracy AMPLIFY beats and the few-shot baseline for the motion-token interface.","marker":"[54]"},{"why":"Track2Act is the competing keypoint-dynamics baseline compared on delta-AUC; unlike AMPLIFY it requires a goal image.","marker":"[53]"},{"why":"BAKU supplies the efficient cross-attention transformer decoder with a Gaussian action head that the inverse dynamics model is built on.","marker":"[62]"},{"why":"ACT contributes the action-chunking and temporal-ensembling scheme used at inference to aggregate action predictions.","marker":"[63]"},{"why":"LIBERO is the benchmark whose five suites ground the few-shot, in-distribution, and zero-action-data generalization claims.","marker":"[66]"},{"why":"Diffusion Policy is the behavior-cloning baseline for the real-world comparison and supplies the action head swapped into the inverse model.","marker":"[68]"},{"why":"AVDC is the video-prediction model that, conditioned on AMPLIFY motion tokens, produces the improved generation metrics.","marker":"[44]"},{"why":"Seer is the full-video-prediction baseline whose generated frames are tracked with CoTracker for the BridgeData accuracy comparison.","marker":"[67]"}],"fun_headline_variants":["Motion tokens from videos boost few-shot robot learning by 1.2-2.2x","Discrete motion priors let robots learn control from action-free video","Video-only pretraining unlocks zero-action-data LIBERO generalization","AMPLIFY: Predicting motion tokens from video to drive robot policies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline stands or falls on the assumption that a re-initialized 20 by 20 grid of 2D keypoint tracks, compressed into 2048 discrete codes, preserves the information needed to choose the right action, and that the point tracker's tracks can be treated as ground truth — even though the inverse model never sees the goal and the paper's Limitations section concedes that 2D tracks leave ambiguity between actions.","fun_headline_variants_meta":{"raw":{"variants":["Motion tokens from videos boost few-shot robot learning by 1.2-2.2x","Discrete motion priors let robots learn control from action-free video","Video-only pretraining unlocks zero-action-data LIBERO generalization","AMPLIFY: Predicting motion tokens from video to drive robot policies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000566,"raw_usage":{"total_tokens":2704,"prompt_tokens":986,"completion_tokens":1718,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":1639}},"tokens_in":602,"tokens_out":1718,"duration_ms":13275,"temperature":1.0,"reasoning_tokens":1639,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:18:35.465170+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a pair of tasks whose keypoint motion is nearly identical over the 16-frame prediction horizon but whose correct actions diverge (for example, two LIBERO-Goal tasks that differ only in which of two objects is the target, filmed so the first 0.8 seconds of motion look alike). If AMPLIFY still solves both at a rate near its reported 60% average, the motion-token interface is more informative than the 2D-ambiguity caveat suggests; if success collapses toward the near-zero inverse-only baseline, the information bottleneck sits in the 2D track representation rather than in the action head.","supporting_citations":[],"review_version":1}