{"id":"2fc77d62-a36a-431e-895a-be2e36b99bc3","arxiv_id":"2608.05706","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LAWM-3D learns 3D-aware latent actions from multi-view human videos by combining VGGT geometric alignment with RGB-depth reconstruction, improving robot world model prediction and generalization.","lead":"This paper presents LAWM-3D, a method that learns 3D-aware action codes from multi-view human videos and uses them to train robot world models. A generalist reader would care because it is a step toward robot simulators that learn from unlabeled video instead of expensive action labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Future-frame appearance leakage is not prevented by the depth objective as described, and no diagnostic measures it; without one, the 3D-awareness causal claim is unproven.","rationale":"The paper's central promise is that the learned latent actions are 3D-aware and motion-centric, and that this is the cause of downstream world-model improvements. For that promise to hold, the model must not be able to copy f_{t+1} into a. The manuscript explicitly states that future-frame leakage is the primary problem it is solving, yet the proposed non-injective RGB-D objective does not, as described, remove the leakage channel: the encoder still sees f_{t+1}, the KL bottleneck is negligible, and the depth target is a deterministic function of the future RGB frame. The Appendix's action-information and linear-probing experiments demonstrate action-relevant content but do not rule out simultaneous appearance content, so they cannot support the stronger causal claim. This is the most load-bearing concern because it attacks the mechanism behind the headline result, not just the quality of a comparison or an evaluation metric. The depth-evaluation circularity and the omitted MVP-LAM comparison are real secondary issues, but even resolving them would not establish that the depth objective prevents leakage. The concern is addressable with a direct diagnostic, and the paper otherwise has broad benchmarking and honest limitations, so rejection is not warranted. Since the reader's CONDITIONAL verdict already hinges on this weakness, my read does not change the verdict.","tokens_in":19984,"tokens_out":8830,"duration_ms":85485,"concrete_test":"Compute I(a; f_{t+1} | f_t) and I(a; f_t) on held-out Ego-Exo4D clips using the same MINE/KSG estimation procedures already used in Appendix B, with f_t and f_{t+1} represented by the LAM's patch-token features to make estimation tractable. If I(a; f_{t+1} | f_t) is a substantial fraction of I(a; f_t), the future frame's content is encoded in a despite the depth objective, and the leakage concern lands; if it is near zero, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the learned latent actions are motion-centric and 3D-aware rather than compressed copies of the future frame. The paper itself identifies future-frame appearance leakage as the key obstacle, but the proposed remedy is not structurally sufficient as described. In Eq. (2), the encoder is q_phi(a | f^v_{t:t+1}), so the future frame is an explicit encoder input. With beta = 1e-6, the KL term imposes essentially no information bottleneck on a. The added depth objective predicts an offline Depth-Anything-V2 depth map d_{t+1}; since d_{t+1} is a deterministic function of f_{t+1} in the training data, any latent that copies f_{t+1} also solves the depth objective. The term 'non-injective' describes the RGB-to-depth map, not the latent code, and the joint loss still contains log p(f_{t+1} | a, f_t), so restricting one output channel does not remove the direct appearance-leakage channel through RGB reconstruction. No experiment in the paper measures I(a; f_{t+1} | f_t) or any equivalent leakage diagnostic. The Appendix B mutual-information and linear-probing analyses show that a contains action-related information, but action relevance and appearance leakage are not mutually exclusive: a can contain both. If leakage persists, the gains in Tables 1-3 may reflect copying future appearance or increased model capacity rather than 3D-aware motion semantics, and the causal attribution in the abstract and conclusion is underdetermined. The Limitations section acknowledges depth noise and VGGT dependence but does not address this leakage question.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LAWM-3D, a latent action model trained on multi-view and monocular human videos with three coupled components: multi-view invariant action tokenization, geometric feature alignment to a pretrained 3D foundation model (VGGT), and an RGB-D joint reconstruction objective claimed to prevent future-frame appearance leakage. The learned latent actions are used to condition a Cosmos-Predict2.5 world model that is pretrained on human videos and post-trained on robot data. Experiments compare latent action models on Ego-Exo4D and Assembly101, evaluate world model video generation on WorldArena across 16 metrics, test OOD generalization on three benchmarks, and include ablations of the proposed components plus appendix analyses of view invariance, mutual information, linear probing, and computational cost.","tokens_in":20269,"tokens_out":5630,"duration_ms":45305,"significance":"If the causal attribution holds, this is a meaningful step toward 3D-aware latent action learning and could improve world models for robotics by exploiting abundant unlabeled human videos. The paper's strengths include broad evaluation across many metrics and OOD benchmarks, careful ablated study of the proposed components and loss weights, analysis of layer-range and alignment-target choices, and explicit computational cost accounting. The appendix's mutual-information estimates with three estimators and linear-probing results across ID/OOD tasks provide useful evidence that the latent space contains action-related structure, and the cross-view transfer and cosine-similarity analyses support view-invariance claims. However, the central claim that the proposed objective prevents future-frame leakage and thereby causes the observed gains is not measured, and several evaluation details, especially the reference for depth metrics, need clarification before the results can be fully interpreted.","major_comments":[{"comment":"The claim that the non-injective RGB-D objective prevents future-frame appearance leakage is not supported by the stated mechanism. In Eq. (2), the encoder remains q_phi(a|f^v_{t:t+1}), so the future frame is an explicit encoder input; with beta = 1e-6 the KL term imposes essentially no information bottleneck; the objective still contains log p(f_{t+1}|a,f_t); and the depth target d_{t+1} is a deterministic function of f_{t+1} in the training data, meaning a latent code that simply copies f_{t+1} also satisfies the depth prediction term. The term 'non-injective' describes the RGB-to-depth mapping, not the latent code. No experiment in the paper measures I(a; f_{t+1} | f_t) or an equivalent leakage diagnostic; the Appendix B mutual-information and linear-probing analyses show that a contains action-related information, but action relevance and appearance leakage are not mutually exclusive. Without such a diagnostic, the causal attribution in the abstract and conclusion that 3D-aware latent actions drive the world-model gains remains underdetermined.","section":"Eq. (2), Section 'Pixel-Agnostic Learning via Depth Map'"},{"comment":"Table 1 lists MVP-LAM in the comparison but does not include any MVP-LAM row, so the claimed superiority over this baseline is not empirically documented. In addition, no table reports error bars, confidence intervals, or significance tests, although the text repeatedly uses 'significantly' (e.g., Abstract and the Comparison of LAMs section). Several margins in Table 1 are small (RGB PSNR 33.38 vs 32.56 against DreamDojo-D), and without variance information the stability of the empirical claim cannot be assessed. Please add the missing baseline or justify its exclusion, and report statistical significance or variance estimates for the central comparisons.","section":"Table 1 and 'Comparison of LAMs'"},{"comment":"The depth metrics are not referenced to a defined ground truth. The LAM is trained with Depth-Anything-V2 pseudo-depth (Implementation Details and Limitations), and if the depth PSNR/SSIM in Table 1 and DepthAcc in Table 2 are computed against the same pseudo-depth, those numbers measure fit to the training label generator, not metric 3D accuracy. This matters because the Limitations section itself states that the depth signals are 'inherently noisy' and provide 'relative rather than metric depth,' while the main text uses the depth results to claim stronger 3D scene understanding. Please specify the reference for each depth metric, define DepthAcc, and if pseudo-depth is used, either add a metric-depth or multi-view-consistency evaluation or temper the 3D-accuracy claims.","section":"Table 1, Table 2, and 'Limitations'"}],"minor_comments":[{"comment":"The notation in Eqs. (3)-(4) is inconsistent with the stated dimensions: y is defined as K x V x P' x D' but the losses sum over p = 1..P; please clarify how VGGT features are aligned to the encoder patch grid and how P' and D' are handled by the projection head.","section":"Eqs. (3)-(4)"},{"comment":"Several column names in Table 2 (JEPASimilarity, DynamicDegree, FlowScore, DepthAcc, Perspectivity) are not defined in the text, and it is not stated whether higher is better for each; please add a metric appendix or table footnote.","section":"Table 2"},{"comment":"The velocity v_i in Eq. (6) is not defined; please specify how velocity is computed from the latent video sequence in the temporal consistency loss.","section":"Eq. (6)"},{"comment":"The author list contains a typo ('Jiale Zhange'), and the reference list duplicates Ye et al. as both 2025a and 2025b with identical titles; please correct these.","section":"References and author list"},{"comment":"The method section states that depth provides 'view-consistent geometric cues,' while the Limitations section notes that the offline depth is noisy, relative, and possibly inconsistent across frames and views; please reconcile these statements.","section":"Method vs Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the experimental effort is substantial. My main reservation is not the architecture itself but the absence of a leakage diagnostic and the ambiguity of the depth references; both are fixable within the scope of a revision. I would not reject on the current evidence, but the causal claim in the abstract should be softened until the leakage question is resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a credible integration of three ideas for latent action learning from multi-view human video: unified view-invariant action tokenization with random view masking, geometric alignment of encoder features to VGGT, and a joint RGB-depth reconstruction objective. The combination is new relative to DreamDojo, UniLACT, and MVP-LAM. What it does well is also substantial: broad evaluation across 16 metrics, OOD generalization tests, and an honest ablation that isolates each component. The negative result in Case1, where naively adding multi-view data alone hurts performance, is a useful finding that motivates the alignment mechanism. The appendices are unusually thorough, including mutual information and linear probing analyses, ablations of layer ranges and loss weights, and comparisons with alternative 3D backbones. That is real evidence of systematic thinking.\n\nThe main soft spot is the leakage claim. The paper says the non-injective RGB-D objective prevents future-frame appearance leakage, but the encoder still takes ft:t+1 as input, beta=1e-6 imposes essentially no bottleneck, and depth d_{t+1} is a deterministic function of f_{t+1} in the training data. A latent that copies the future frame also solves the depth objective. The paper never directly measures the information shared between the latent action and f_{t+1} given f_t, or any equivalent diagnostic. The appendix analyses show action-related information, but action relevance and appearance leakage are not mutually exclusive; a latent can contain both. So the causal attribution, that 3D awareness drives the downstream gains, is underdetermined. This is not fatal to the empirical contribution, since the method outperforms baselines on many metrics and the OOD gains suggest something beyond copying, but the mechanism needs a direct test.\n\nOther soft spots are minor: no error bars or significance tests in the main tables; the depth reference in Table 1 is unspecified, which matters for interpreting those PSNR/SSIM numbers; MVP-LAM is absent from the headline LAM comparison, appearing only in appendix analyses; and no code or checkpoints are released, limiting verification.\n\nWho this is for: anyone working on latent action models, world models for manipulation, or self-supervised action representation from video. It deserves a serious referee. I would send it to review, with the leakage diagnostic as the key requested revision.\n\nMy recommendation: accept conditional on a direct leakage measurement, such as input-invariant probing or mutual information with future appearance, plus clarification of the depth references.","headline":"A solid, broadly evaluated integration of multi-view, depth, and geometric alignment for latent actions, but the paper's central leak-prevention claim needs a direct test.","tokens_in":20870,"tokens_out":1975,"would_cite":true,"duration_ms":17937,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that latent action models can be made 3D-aware, and that this is what lets robot world models generalize across scenes.","keywords":["latent action models","world models","3D-aware representations","multi-view video learning","robot manipulation","RGB-D reconstruction","geometric alignment","video generation"],"falsifier":"Give the trained encoder only the current frame $f_t$ (with $f_{t+1}$ withheld) and measure how well the world model still reconstructs the future; or train a linear probe to decode $f_{t+1}$'s appearance from $a$ alone. If future texture survives in $a$, the leakage-prevention claim is false and the attribution of downstream gains to 3D awareness is unsupported.","tokens_in":19747,"feed_emoji":"🤖","tokens_out":6068,"duration_ms":45453,"temperature":0.7,"pith_summary":"This paper argues that simply adding multi-view video to latent action models is not enough to give the learned actions 3D awareness; viewpoint discrepancies and future-frame appearance leakage get in the way. It proposes LAWM-3D, a latent action autoencoder with three coupled designs—multi-view invariant tokenization, geometric alignment to a pretrained 3D foundation model, and a non-injective RGB-D reconstruction objective—that together force the latent action to encode view-invariant, geometrically meaningful motion. Trained on large-scale human videos and fine-tuned on robot data, the resulting 3D-aware latent actions improve a downstream diffusion world model in generation quality, physical consistency, controllability, and out-of-distribution generalization. If correct, this gives a route to robot world models that learn motion semantics from unlabeled video rather than from expensive action annotations.","feed_headline":"3D-aware latent actions push robot world models past 2D baselines","feed_subtitle":"View-masked tokens, VGGT geometry alignment, and RGB-D decoding stop appearance leakage and give rollouts real 3D motion cues.","key_machinery":"The central object is the 3D-aware latent action $a$, a 32-dimensional vector produced by a multi-view spatio-temporal Transformer encoder from a set of synchronized observations $F_{t:t+1}$. Three coupled mechanisms carry the argument: (1) multi-view invariant tokenization, which randomly masks views during training and aggregates the action tokens by mean pooling so the same motion produces the same action from any viewpoint; (2) geometric representation alignment, a cosine and scale loss that anchors the encoder's intermediate layers ($h=[6,11]$) to the frozen VGGT foundation model's view-consistent 3D features; and (3) a non-injective RGB-D joint reconstruction objective, where the decoder predicts both future frame and depth from current RGB-D input, with depth computed offline and never given to the encoder, so the latent action cannot cheat by copying future appearance. The claim is that these together make $a$ a pure, geometrically grounded motion code that a diffusion world model can condition on.","core_discovery":"On the paper's own terms, the central discovery is that a latent action representation learned from unlabeled video can be made 3D-aware, and that this is what unlocks world-model gains. Naive multi-view training fails because the encoder can copy future-frame RGB content into the latent action and because different cameras disagree on appearance. The paper's three mechanisms close exactly those holes: random view masking plus cross-view aggregation makes the action token view-invariant; aligning intermediate encoder features with VGGT injects explicit cross-view geometric correspondences; and decoding jointly into RGB and depth—with depth unavailable to the encoder—removes the appearance shortcut. Through a two-stage human-video pretraining and robot post-training pipeline, LAWM-3D reports state-of-the-art results across 16 benchmark metrics and on out-of-distribution rollouts, with the largest gains in motion quality and 3D accuracy.","pith_inferences":["A direct leakage probe—decoding future texture from $a$ alone with the current frame fixed—would turn the paper's leakage-prevention claim into a measurable quantity; the paper does not report such a probe, but this test is a natural extension of its analysis.","The alignment target need not be VGGT specifically; the paper's own ablations show DUSt3R and Depth-Anything-V2 also help, so the mechanism is geometric priors in general. One could extend the approach to single-view data by generating pseudo-multi-view pairs from a depth-based renderer, which would scale pretraining beyond synchronized multi-view corpora.","Because the latent action is view-invariant, the same learned action space could serve as a shared control interface across camera rigs in real deployment, letting a policy condition on actions extracted from whatever camera is available.","The world-model-based policy selection experiments suggest imagined returns correlate strongly with real task performance (Spearman $\\rho=0.952$); an untested but plausible consequence is that the same selection mechanism could rank policies in the wild where no reward signal is available."],"forward_implications":["If latent actions are truly 3D-aware and view-invariant, a world model conditioned on them should produce rollouts that stay physically consistent longer, since the action code tracks 3D motion rather than 2D pixels.","The two-stage recipe (human-video pretraining plus robot post-training) becomes more data-efficient: unlabeled multi-view human videos can substitute for expensive robot action annotations in world-model pretraining.","Because the latent action model is frozen and not needed at rollout time, the 3D awareness comes at negligible inference cost, so the approach can be dropped into existing world-model backbones.","The learned action space should transfer across viewpoints and embodiments, meaning a single action vector extracted from one camera can drive prediction from another camera's observation.","Out-of-distribution generalization in novel scenes and interactions should be stronger than for 2D latent action models, since geometry alignment anchors features to scene structure rather than appearance."],"supporting_citations":[{"why":"DreamDojo is the baseline latent action model and world-model pipeline this paper builds on and compares against; it supplies the β-VAE formulation and the post-training protocol.","marker":"(Gao et al. 2026a)"},{"why":"VGGT is the frozen 3D foundation model whose view-consistent features serve as the target for the geometric alignment loss.","marker":"(Wang et al. 2025)"},{"why":"Geometry Forcing provides the inspiration for aligning learned video features with pretrained 3D foundation models to enforce geometric consistency.","marker":"(Wu et al. 2026)"},{"why":"Depth-Anything-V2 is the offline monocular depth estimator that generates the depth maps used in the non-injective RGB-D reconstruction objective.","marker":"(Yang et al. 2024)"},{"why":"Ego-Exo4D is the largest multi-view human manipulation video dataset used for pretraining the latent action model.","marker":"(Grauman et al. 2024)"},{"why":"Assembly101 provides a second synchronized multi-view dataset with egocentric and static cameras for joint training.","marker":"(Sener et al. 2022)"},{"why":"EgoDex supplies the single-view egocentric dexterous manipulation data used in joint training and in out-of-distribution generalization evaluation.","marker":"(Hoque et al. 2025)"},{"why":"WorldArena is the robot world-model benchmark whose 16 metrics across six dimensions are used to score generation quality and controllability.","marker":"(Shang et al. 2026)"}],"fun_headline_variants":["LAWM-3D stops appearance leakage to make latent actions truly 3D-aware","View-masked tokens and VGGT alignment give robot world models 3D muscles","Single-view latent actions fail to see depth: LAWM-3D adds geometric anchors","Multi-view alone won't do: LAWM-3D couples view-masking with VGGT geometry","LAWM-3D: depth-aware latent actions from videos generalize to robots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the non-injective RGB-D objective actually stops the encoder from copying the future frame's appearance into the latent action, so that the measured downstream gains are caused by 3D awareness rather than by information still leaking into $a$.","fun_headline_variants_meta":{"raw":{"variants":["LAWM-3D stops appearance leakage to make latent actions truly 3D-aware","View-masked tokens and VGGT alignment give robot world models 3D muscles","Single-view latent actions fail to see depth: LAWM-3D adds geometric anchors","Multi-view alone won't do: LAWM-3D couples view-masking with VGGT geometry","LAWM-3D: depth-aware latent actions from videos generalize to robots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000577,"raw_usage":{"total_tokens":2773,"prompt_tokens":1047,"completion_tokens":1726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":1613}},"tokens_in":663,"tokens_out":1726,"duration_ms":11762,"temperature":1.0,"reasoning_tokens":1613,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:35:42.590668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the trained encoder only the current frame $f_t$ (with $f_{t+1}$ withheld) and measure how well the world model still reconstructs the future; or train a linear probe to decode $f_{t+1}$'s appearance from $a$ alone. If future texture survives in $a$, the leakage-prevention claim is false and the attribution of downstream gains to 3D awareness is unsupported.","supporting_citations":[],"review_version":2}