{"id":"83d0483b-be7d-49ca-ab28-e3362c28f5f0","arxiv_id":"2508.09822","paper_version":4,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PAR combines video generation with physical tokens for frames and actions, enabling manipulation without action pretraining: 100% on PushCube and matching action-pretrained baselines elsewhere.","lead":"A new model, PAR, lets robots learn manipulation by watching videos, without first training on robot actions. It predicts future frames and control actions together, and reports perfect success on a simple pushing task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Concern: PAR's action generation may rely on action supervision in the physical tokens rather than on video-pretrained physical understanding; an ablation is needed.","rationale":"The reader's weakest assumption is that video pretraining encodes sufficient physical knowledge for manipulation and that tokenization does not lose action-relevant information. My concern sharpens this: the physical tokens themselves contain action labels, so the model has direct access to action supervision during training. This means the observed success could be attributed to the action component of the tokens rather than to transferred world knowledge. This is a load-bearing issue because if the action supervision is doing the work, the 'without action pretraining' claim is misleading—the model is still trained with per-step action labels, just not on a separate large-scale action dataset. The proposed ablation is a concrete, feasible check that would settle whether video pretraining is necessary. However, since the full text is unavailable and the abstract alone cannot resolve this, the existing UNVERDICTED verdict is appropriate. I partially agree with the reader because we both target the sufficiency of video pretraining, but my concern is more specific about the confound with action tokens.","tokens_in":675,"tokens_out":2773,"duration_ms":35282,"concrete_test":"Obtain the full text and run two ablations: (1) Train PAR from scratch on the manipulation data (randomly initialized video backbone) with the same physical-token architecture and compare PushCube success and video-prediction accuracy; if performance remains near 100%, the video pretraining is not load-bearing for the result. (2) During training, mask the action component in the input token sequence so the model must predict actions solely from visual frames, then evaluate action accuracy; if this collapses, the model is not deriving actions from physical understanding but merely copying action supervision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that video pretraining supplies world knowledge that enables consistent action trajectories without action pretraining. However, the abstract states that physical tokens combine frames and actions, meaning the model is trained with explicit action labels as part of its token sequence. This creates a potential confound: the model may simply learn a mapping from the observed frame(s) to the action label via the action supervision embedded in the tokens, and the video pretraining may contribute little to action generation. The reported 100% success on PushCube, a task with a simple, near-deterministic policy, is consistent with overfitting to action labels rather than leveraging physical dynamics. If this is the case, the headline claim of transferring world knowledge from video pretraining is substantially weakened, and the method reduces to a supervised policy with a video-prediction auxiliary objective. The abstract provides no ablation separating the contribution of video pretraining from that of action supervision, so the central mechanism is unproven.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a Physical Autoregressive Model (PAR) for robotic manipulation. PAR builds on autoregressive video generation models and introduces 'physical tokens' that combine frames and actions, representing the joint evolution of robot and environment. The model uses a DiT-based de-tokenizer to handle continuous tokens, and incorporates a causal mask with inverse kinematics, parallel training, and KV-cache for efficiency. The paper claims that, without any action pretraining, PAR achieves a 100% success rate on the PushCube task in ManiSkill, matches action-pretrained baselines on other tasks, and produces accurate future-video predictions with tightly aligned action trajectories.","tokens_in":919,"tokens_out":2282,"duration_ms":27233,"significance":"If the results hold, the paper would demonstrate a valuable transfer of world knowledge from video-pretrained autoregressive models to robotic manipulation, potentially reducing reliance on expensive action-labeled data. The core idea is concrete and testable: represent frames and actions together in a token sequence and leverage video pretraining to predict both. The reported 100% success on PushCube is a sharp, falsifiable claim. However, the evidence presented in the abstract is not sufficient to assess the central mechanism: no ablations separate the contribution of video pretraining from that of action supervision, and no statistical details support the quantitative claims. The significance of the direction is real, but the current support is thin.","major_comments":[{"comment":"This is the central confound for the headline claim. The model is trained with action labels as part of its input token sequence, so the action supervision is embedded in the training objective. The 100% PushCube success could stem from the model learning a supervised frame-to-action mapping rather than from video-pretrained physical understanding. An ablation must be provided that isolates the video pretraining contribution: (i) train PAR from scratch without video pretraining; (ii) freeze the video-pretrained weights and train only action-related components; (iii) remove action tokens from the input and predict actions only as outputs. Without this, the claim of 'without action pretraining' is not established.","section":"Abstract — 'physical tokens combine frames and actions'"},{"comment":"The abstract reports a 100% success rate on PushCube and says PAR 'matches the performance' of action-pretrained baselines, but gives no number of trials, seeds, error bars, or any quantitative comparison table. It is impossible to judge whether 100% is one success out of one episode or a robust result. Report mean success with standard error over multiple seeds and at least 10–50 episodes, and compare against a non-pretrained baseline and a standard behavior-cloning agent to contextualize the gains.","section":"Abstract — experimental reporting"},{"comment":"These claims lack defined metrics. 'Accurate' and 'tightly aligned' are not measurable without quantitative evaluation. Specify the video prediction metrics (e.g., FVD, PSNR, or LPIPS) and action alignment metrics (e.g., mean squared error between predicted and ground-truth actions, or open-loop success rate when using predicted actions). Without these, the central evidence for the method's physical understanding is incomplete.","section":"Abstract — 'accurate video prediction' and 'tightly aligned action trajectories'"}],"minor_comments":[{"comment":"The terms 'physical tokens' and 'DiT-based de-tokenizer' are introduced without definitions. Since the abstract is the only available text, these should be briefly characterized, e.g., what exactly is tokenized and how the de-tokenizer reconstructs continuous values.","section":"Abstract — definitions"},{"comment":"The abstract says the model leverages 'video pretraining' but does not specify the pretrained model or its training data. State which backbone is used (e.g., an autoregressive video model) and whether the pretraining data includes robotic or manipulation videos, since this affects the interpretation of 'world knowledge'.","section":"Abstract — pretraining details"},{"comment":"The phrase 'the project page is here:' is informal; use a standard citation or reference to the URL instead.","section":"Abstract — presentation"}],"recommendation":"major_revision","confidential_remarks":"This review is based solely on the abstract because the full text was not available. The stress-test concern is valid: action supervision is embedded in the physical tokens, so the central claim of transferring world knowledge from video pretraining is not yet supported without ablations. The required experiments are within the manuscript's scope and should be feasible to add. The lack of statistical detail also needs to be corrected. I recommend major revision rather than rejection because the proposed direction is plausible and the missing evidence is concrete and obtainable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is abstract-only in front of me, so treat everything as provisional. The central idea — physical tokens that interleave frames and actions inside an autoregressive video model, letting the model reuse video-pretrained representations for manipulation without action pretraining — is a real and plausible direction. The authors also make sensible engineering choices: continuous tokens with a DiT de-tokenizer to avoid quantization loss, a causal mask with inverse kinematics, parallel training, KV-cache. If the reported results hold, this is a useful step for the video-prediction-for-control line.\n\nThe soft spot is not hidden: the abstract says physical tokens combine frames and actions. That means action labels are in the training sequence. The headline claim is that video pretraining supplies the physical understanding, but the model could simply be learning a mapping from frames to actions via that supervision, with the video generation acting as a regularizer. The 100% success on PushCube, a near-deterministic task, is exactly the setting where a supervised policy can look perfect. Without an ablation that removes or down-weights the action supervision, or isolates the contribution of video pretraining, the central mechanism is unproven. The reported 'matches action-pretrained baselines' also lacks trial counts and error bars, which is a significant omission for a benchmark where success rates can be noisy.\n\nThat said, none of this is fatal from the abstract alone. The concern is a confound, not a contradiction; the authors may well have the ablation in the full text. The idea is concrete enough that I'd want to see the experiments. If they show that video pretraining is actually load-bearing, this is a solid venue paper.\n\nMy recommendation: send it to peer review, but push hard for an ablation isolating action supervision from video pretraining, and for statistical detail on all reported success rates. This is a paper that a serious referee can evaluate; it shouldn't be desk-rejected on the abstract alone.","headline":"Abstract-only, but the physical-token idea is plausible; the central confound is whether action supervision rather than video pretraining does the real work.","tokens_in":1323,"tokens_out":1471,"would_cite":false,"duration_ms":16121,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PAR claims that a video-autoregressive model can perform robotic manipulation without any action pretraining, using physical tokens that interleave frames and robot actions, and supports the claim with 100% success on PushCube while matchin","keywords":["robotic manipulation","autoregressive video generation","physical tokens","action pretraining","world knowledge","continuous de-tokenization","ManiSkill benchmark","video prediction"],"falsifier":"Train PAR with randomly initialized video-model weights on the same data and token pipeline: the method's framing predicts that PushCube success falls far below 100%, isolating the contribution of video world knowledge. Separately, replace the continuous DiT de-tokenizer with discrete tokens; if action-video alignment degrades, the quantization explanation is supported.","tokens_in":622,"feed_emoji":"🤖","tokens_out":9628,"duration_ms":79453,"temperature":0.7,"pith_summary":"The paper tries to establish that robotic manipulation does not need a separate action-pretraining stage: a large autoregressive video model already carries enough physical knowledge to generate both the next video frame and the next robot action. To do this, PAR builds physical tokens that combine frames and actions, and predicts them with a single next-token objective; a DiT-based de-tokenizer turns continuous tokens back into frames and actions, which reduces quantization errors and lets the two predictions reinforce each other. On the ManiSkill benchmark, PAR reports 100% success on PushCube, matches the performance of action-pretrained baselines on the other tasks, and produces future videos whose action trajectories line up tightly. The reason to care is that manipulation data is scarce, so a route that transfers world knowledge from video pretraining could unlock a vast existing video corpus for robot skill learning.","feed_headline":"Video pretraining alone: 100% success, zero action labels.","feed_subtitle":"A video-pretrained transformer also issues robot actions, matching baselines that trained with action labels.","key_machinery":"The load-bearing machinery is the physical token, a representation that combines an image frame and its associated robot action into one token so that the autoregressive video-pretraining objective becomes a joint model of the robot and its environment. A DiT-based diffusion de-tokenizer decodes these tokens as continuous values into frames and actions, avoiding the quantization errors of discrete tokens and letting frame prediction and action prediction share one learning signal. A causal mask with inverse kinematics couples each generated action to the robot's observed state, while parallel training and the KV-cache mechanism supply the efficiency.","core_discovery":"The central claim of PAR is that a single autoregressive model, initialized from video pretraining, can jointly generate future video frames and consistent robot action sequences without any action pretraining. Physical tokens combine the frame and the action at each time step, so the next-token objective becomes a shared world-model and policy objective. The DiT-based de-tokenizer decodes these tokens as continuous values, which mitigates quantization errors and enables mutual enhancement between frame prediction and action prediction. A causal mask with inverse kinematics keeps generated actions consistent with the robot state, and parallel training plus the KV-cache make the scheme practi","pith_inferences":["A direct test of the paper's worldview: replacing the video-pretrained weights with random initialization on the same token pipeline should collapse PushCube success; if it does not, the claimed transfer of world knowledge is not the true cause.","The physical-token scheme is generic, so it could be lifted from manipulation to other sequential decision problems — navigation, multi-agent coordination, or dexterous hand control — where state and action share one temporal dynamic.","Contact-rich manipulation where the pixels barely change (in-hand reorientation, soft-body deformation) is a plausible boundary case: video pretraining encodes appearance well but may encode contact forces poorly, a gap the paper does not examine.","If world knowledge is the decisive factor, scaling video pretraining in diversity and resolution should improve manipulation performance across embodiments without extra action data, a prediction that goes beyond the reported benchmark results."],"forward_implications":["If video pretraining supplies the needed physical dynamics, new manipulation skills can be learned with little or no action-labelled data.","Frame prediction and action generation become the same next-token task, so video forecasting and robot control stop being separate problems.","Continuous-token decoding keeps quantization errors from accumulating, which helps long-horizon action trajectories stay aligned with the predicted video.","Parallel training and the KV-cache make action generation fast enough that the approach is closer to real-time robot control."],"supporting_citations":[],"fun_headline_variants":["Video pretraining alone achieves 100% success on PushCube","No action labels? Video pretraining still matches action-trained baselines","Physical tokens merge video and action, no action pretraining required","Autoregressive video model doubles as robot policy without action data"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing assumption is that the physical understanding needed for manipulation is already stored in video pretraining, so that physical tokens built from frames can produce correct actions without ever training on action labels; if that knowledge is missing, or is lost when frames are tokenized, the success on PushCube and the matches to action-pretrained baselines would not hold.","fun_headline_variants_meta":{"raw":{"variants":["Video pretraining alone achieves 100% success on PushCube","No action labels? Video pretraining still matches action-trained baselines","Physical tokens merge video and action, no action pretraining required","Autoregressive video model doubles as robot policy without action data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0004,"raw_usage":{"total_tokens":1914,"prompt_tokens":719,"completion_tokens":1195,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1123}},"tokens_in":463,"tokens_out":1195,"duration_ms":12627,"temperature":1.0,"reasoning_tokens":1123,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:45:52.892522+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train PAR with randomly initialized video-model weights on the same data and token pipeline: the method's framing predicts that PushCube success falls far below 100%, isolating the contribution of video world knowledge. Separately, replace the continuous DiT de-tokenizer with discrete tokens; if action-video alignment degrades, the quantization explanation is supported.","supporting_citations":[],"review_version":1}