{"id":"b4dec31f-f239-4827-8ebb-351a5363f810","arxiv_id":"2607.06559","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":8,"one_line_summary":"A tri-branch diffusion model co-generates RGB, depth, and optical flow from a single RGB-D image, and an inverse dynamics head on its internal latents achieves state-of-the-art bimanual manipulation success rates.","lead":"This paper builds a robot world model that jointly predicts future video, depth, and motion from a single image, then uses those predictions to control a dual-arm robot. It matters because grounding robot actions in 3D geometry rather than 2D pixels could improve manipulation precision.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The SOTA claim against π0/π0.5 is potentially confounded by embodiment mismatch — the paper never specifies whether these VLA foundation models were fine-tuned on the TIANJI/WUJI platform (54-DOF dexterous hands) or run zero-shot on an unfamiliar robot.","rationale":"The reader identified a real ablation confound (ResNet-18 vs. large backbone in the 'w/o RynnWorld-4D' row), but this is partially addressed by the 'RGB' ablation in Table 5, which uses the same RynnWorld-4D backbone with only RGB latents. The progression RGB → RGB+Depth → RGB+DF within the same backbone does isolate the contribution of 4D modalities to a reasonable degree, and even the RGB-only features benefit indirectly from cross-modal training (the JA modules shape all branches during world model training). So the reader's concern is valid but not the most load-bearing issue. The more critical concern is the fairness of the π0/π0.5 comparison. These are VLA foundation models designed for specific embodiments. The TIANJI M6 + WUJI Hand is a 54-DOF dual-arm dexterous system that these models have almost certainly never encountered. If they were not properly fine-tuned on this platform, their low scores reflect embodiment mismatch rather than a fundamental limitation of 2D representations — which would undermine the paper's argument that 'predictive 4D representations are essential for tasks requiring precise temporal and spatial coordination where pure generative 2D priors fall short.' Additionally, with n=35 trials, differences like 65.71% vs 57.14% (23/35 vs 20/35) are not statistically significant (Fisher's exact p≈0.44), yet no variance or confidence intervals are reported. The CONDITIONAL verdict is appropriate: the tri-branch 4D world model and inverse dynamics policy are legitimate engineering contributions with reasonable ablation evidence, but the SOTA claim against foundation models is not well-supported without clarifying the baseline training protocol. The paper should either specify that π0/π0.5 were fine-tuned with matched data, or soften the claim to focus on the internally valid ablation results.","tokens_in":23825,"tokens_out":5211,"duration_ms":387305,"concrete_test":"Fine-tune π0.5 on the same 200 episodes/task with the TIANJI M6 + WUJI Hand platform using the authors' teleoperation data, and re-evaluate on all six tasks. If π0.5's success rate on Hand-over or Lid Placement increases by more than 15 percentage points, the SOTA claim weakens significantly. At minimum, the paper must report whether baselines were zero-shot or fine-tuned, with matching episode counts and training epochs.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim of SOTA performance rests on outperforming π0 (Black et al., 2024) and π0.5 (Intelligence et al., 2025) on tasks involving a 20-DOF WUJI dexterous hand (54 total DOF). π0.5 achieves 0% on Hand-over and 37.14% on Lid Placement, which the paper attributes to 'pre-training data predominantly biased towards parallel-jaw grippers' (Sec. 4.3). However, the paper never specifies whether π0/π0.5 were fine-tuned on the same 200 episodes/task used for RynnWorld-4D-Policy, or run zero-shot. If zero-shot, the comparison is fundamentally unfair — these models were never trained on this embodiment, and their low scores reflect embodiment mismatch, not a limitation of 2D representations. Even if fine-tuned, the absence of any training protocol for baselines makes it impossible to assess whether data and compute are controlled. The reader's ablation concern (ResNet-18 vs. large backbone) is partially mitigated by the 'RGB' row in Table 5, which uses the same RynnWorld-4D backbone with only RGB latents and shows a clear progression: RGB (77.14 on Dual Picking) → RGB+Depth (91.43) → full RGB-DF (94.29). This isolates the 4D modality contribution within the same backbone reasonably well. The more fragile assertion is the SOTA claim against foundation models.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper introduces RynnWorld-4D, a 4D embodied world model that co-generates synchronized RGB, depth, and optical flow (RGB-DF) sequences from a single RGB-D image and text instruction using a tri-branch diffusion transformer. The authors also curate Rynn4DDataset 1.0 (254.4M frames) with pseudo-labels for depth and flow, and propose RynnWorld-4D-Policy, an inverse dynamics head that consumes the frozen world model's internal 4D latents in a single forward pass to generate robotic actions. The system is evaluated on a 54-DOF bimanual dexterous manipulation platform (TIANJI M6 + WUJI Hand) across six real-world tasks, demonstrating strong performance in spatially precise tasks.","tokens_in":24178,"tokens_out":1231,"duration_ms":129012,"significance":"The paper presents a well-executed, large-scale effort integrating generative world modeling with robotic control. The tri-branch architecture with Joint Cross-Modal Attention and the phased training strategy (modality adaptation, frozen-backbone joint attention, full SFT) is technically sound. The provision of a large-scale dataset, reproducible code, and model weights is a notable strength. The core hypothesis—that jointly modeling RGB, depth, and flow within one diffusion loop provides predictive 4D features closer to robot actions than 2D pixel latents—is well-motivated and supported by the within-backbone ablation (Table 5). The system demonstrates impressive real-world dexterous manipulation results.","major_comments":[{"comment":"§4.3, Table 5: The headline claim of 'state-of-the-art performance' against foundation models π0 and π0.5 is potentially confounded by an embodiment mismatch. The paper evaluates these models on a 54-DOF dexterous hand system (TIANJI/WUJI), but never specifies whether π0/π0.5 were fine-tuned on the 200 episodes/task used for RynnWorld-4D-Policy or run zero-shot. If zero-shot, the low scores (e.g., π0.5 achieving 0% on Hand-over) reflect embodiment mismatch rather than a limitation of 2D representations. The paper must explicitly state the training protocol for these baselines to ensure data and compute are controlled. Without this, the SOTA claim is unsubstantiated.","section":null},{"comment":"§4.3, Table 5: The 'w/o RynnWorld-4D' ablation uses a ResNet-18 backbone, which is drastically weaker in scale and pretraining than the proposed 5B-parameter diffusion transformer. This makes it impossible to isolate whether the 4D representation or the model scale drives the gains. While the within-backbone ablations (RGB vs. RGB+Depth vs. full RGB-DF) partially mitigate this by isolating the 4D modality contribution within the same backbone, the comparison against ResNet-18 conflates representation quality with model capacity. The authors should acknowledge this confound or provide a more comparably-scaled 2D baseline.","section":null},{"comment":"§4.2, Table 4: The world model evaluation uses only 50 held-out video sequences. This is a very small sample size for reporting metrics with three decimal places (e.g., AbsRel=0.310, AEPE=0.170) and claiming significant advantages over baselines. The variance of these metrics is unknown and could be substantial. The authors should report confidence intervals or standard deviations, and ideally expand the test set to provide a more robust evaluation.","section":null}],"minor_comments":[{"comment":"§3.5: The claim of 'high-frequency, closed-loop control' at ~9 Hz is somewhat overstated given that the system executes 10 actions open-loop over ~1.1s while the next plan is computed. The distinction between planning frequency (~0.9 Hz) and effective control frequency (~9 Hz) is noted but could be clarified to avoid misleading impressions about the system's reactivity.","section":null},{"comment":"§3.1: The depth quantization to 8-bit grayscale (I = ⌊d/dmax × 255⌋) with a global range of [0.0, 5.0] meters may lose fine-grained depth precision, which is critical for the dexterous manipulation tasks evaluated. The authors should discuss whether this quantization limits the geometric accuracy of the policy.","section":null},{"comment":"Table 4: The AEPE metric is computed in 'normalized RGB color space' between color-coded flow maps rather than in the native flow space. This is an unusual choice that may not accurately reflect the true optical flow error. The authors should justify this decision or provide native-space AEPE for comparison.","section":null},{"comment":"§4.1.2: The choice of diffusion timestep t=500 for feature extraction is stated without justification. Was this value empirically optimized? A brief ablation or rationale would strengthen the paper.","section":null},{"comment":"Figures: Several figures (e.g., Fig. 4, 6) are dense and could benefit from larger fonts or clearer labeling of components for readability.","section":null}],"recommendation":"major_revision","confidential_remarks":"The central technical contribution (the tri-branch 4D world model and its integration with a policy head) is solid and represents a substantial engineering effort. However, the SOTA claim against π0/π0.5 is the weakest link—if those baselines were run zero-shot on an unfamiliar 54-DOF embodiment, the comparison is fundamentally unfair and the claim cannot stand. This must be clarified or the claim must be retracted. If the authors can confirm that baselines were properly fine-tuned, or if they soften the SOTA claim to focus on the 4D representation's benefits (which is well-supported by the within-backbone ablations), the paper could be acceptable. The small world model evaluation sample size (50 sequences) is also a concern but is secondary to the baseline comparison issue."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. All three major comments identify legitimate gaps in our experimental presentation that we will address in the revision. Below we respond point by point.","responses":[{"response":"The referee is correct that the manuscript does not explicitly state the training protocol for π0 and π0.5, and this omission undermines the SOTA claim. To clarify: π0 and π0.5 were fine-tuned on the same per-task demonstration data (200 episodes/task) used for RynnWorld-4D-Policy, using the official fine-tuning recipes released by the respective authors. We adapted the action space mapping from the 54-DOF TIANJI/WUJI system to the format expected by each model. However, we acknowledge that even with fine-tuning, these models were pretrained predominantly on parallel-jaw gripper data and lack dexterous hand priors, which introduces a pretraining-data mismatch that is distinct from a pure representation-quality comparison. We agree this distinction must be made explicit. In the revision we will: (1) add a clear description of the fine-tuning protocol for all baselines in §4.2; (2) reframe the comparison as evaluating 2D-pretrained VLA models after task-specific fine-tuning on a novel embodiment, rather than as a controlled representation comparison; and (3) soften the 'state-of-the-art' language to reflect that the comparison is between our 4D-representation pipeline and 2D-pretrained VLAs adapted to this embodiment, not a controlled isolation of representation quality alone.","revision_made":"yes","referee_comment":"§4.3, Table 5: The headline claim of 'state-of-the-art performance' against foundation models π0 and π0.5 is potentially confounded by an embodiment mismatch. The paper evaluates these models on a 54-DOF dexterous hand system (TIANJI/WUJI), but never specifies whether π0/π0.5 were fine-tuned on the 200 episodes/task used for RynnWorld-4D-Policy or run zero-shot. If zero-shot, the low scores reflect embodiment mismatch rather than a limitation of 2D representations. The paper must explicitly state the training protocol for these baselines to ensure data and compute are controlled."},{"response":"This is a fair criticism. The ResNet-18 ablation was intended to show that static 2D features are insufficient compared to predictive 4D latents, but it does conflate model scale with representation quality. We agree that this comparison alone cannot isolate the contribution of the 4D representation from the contribution of model capacity. The within-backbone ablations (RGB vs. RGB+Depth vs. RGB+Optical Flow vs. full RGB-DF, all using the same 5B RynnWorld-4D backbone) are the controlled comparisons that isolate the modality contribution, and we will emphasize these as the primary evidence for the value of 4D representations. In the revision we will: (1) explicitly acknowledge the scale confound in the ResNet-18 comparison; (2) reposition the within-backbone ablations as the main evidence for the 4D representation hypothesis; and (3) note that a comparably-scaled 2D-only baseline (e.g., the same 5B backbone trained on RGB-only video) is a natural next step, and we will add the RGB-only within-backbone result from Table 5 as the closest available controlled comparison. We will not claim that the ResNet-18 comparison isolates representation quality.","revision_made":"yes","referee_comment":"§4.3, Table 5: The 'w/o RynnWorld-4D' ablation uses a ResNet-18 backbone, which is drastically weaker in scale and pretraining than the proposed 5B-parameter diffusion transformer. This makes it impossible to isolate whether the 4D representation or the model scale drives the gains. The authors should acknowledge this confound or provide a more comparably-scaled 2D baseline."},{"response":"The referee is correct that 50 sequences is a small sample for the precision of reporting and the strength of the claims made. We will address this in two ways. First, we will expand the held-out test set from 50 to at least 200 sequences to provide more robust estimates. Second, we will report standard deviations (or 95% confidence intervals) for all metrics in Table 4. We will also adjust the number of reported decimal places to be consistent with the uncertainty. We acknowledge that without variance estimates, the current claims of 'significant advantages' are not statistically substantiated. The revision will include the expanded evaluation with uncertainty quantification, and we will temper any comparative language accordingly where overlaps in confidence intervals exist.","revision_made":"yes","referee_comment":"§4.2, Table 4: The world model evaluation uses only 50 held-out video sequences. This is a very small sample size for reporting metrics with three decimal places and claiming significant advantages over baselines. The authors should report confidence intervals or standard deviations, and ideally expand the test set."}],"tokens_in":23472,"tokens_out":1620,"duration_ms":87634,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The core contribution is a tri-branch diffusion model that co-generates RGB, depth, and optical flow within a single denoising loop, then feeds the frozen 4D latents into a lightweight inverse dynamics head for policy learning. The projective RGB-DF representation is a clean design choice — it stays in 2D-aligned format so you inherit video diffusion priors, while depth + flow back-project to explicit 3D scene flow. The phased training (modality adaptation → frozen-backbone joint attention → full SFT) is well-motivated, and the ablations in Table 4 show each component earns its keep. Shipping code, models, and a 254M-frame dataset is real evidence. The within-backbone ablation in Table 5 (RGB → RGB+Depth → full RGB-DF on the same RynnWorld-4D encoder) does a reasonable job isolating the 4D modality contribution: the progression on Dual Picking (77 → 91 → 94) is clean enough to support the claim that depth and flow latents add value beyond RGB alone. The reader's concern about the ResNet-18 baseline confound is valid for that specific row, but the same-backbone ablation rows partially address it. The stress-test concern about π0/π0.5 is the real problem. The paper never specifies whether these models were fine-tuned on the TIANJI/WUJI platform (54-DOF dexterous hands) or run zero-shot. If zero-shot, the comparison is unfair — π0.5 scoring 0% on Hand-over likely reflects embodiment mismatch, not a limitation of 2D representations. The paper attributes the gap to gripper bias in pretraining data, but without a training protocol for baselines, this is an assertion, not evidence. This undercuts the SOTA framing. Minor issues: t=500 for feature extraction is unjustified, no variance over 35 trials, and the 1.1s planning latency with 10-action open-loop execution stretches the closed-loop claim. The world model evaluation on 50 sequences is thin. These are fixable. The central engineering contribution — a 4D world model whose latents demonstrably help policy learning on the same backbone — holds up. This paper is for researchers working on world models for manipulation and anyone interested in geometric representations for policy learning. It deserves a serious referee. The referee should push hard on the baseline comparison protocol and request either fine-tuned π0/π0.5 on the same embodiment or a clear acknowledgment that the SOTA claim is conditional on embodiment mismatch.","headline":"Solid engineering with a real 4D representation; the SOTA claim against π0/π0.5 is the weak point.","tokens_in":24743,"tokens_out":603,"would_cite":false,"duration_ms":51903,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"4D world model beats 2D policies on bimanual robot tasks","keywords":[],"falsifier":"A single-branch RGB-only transformer with the same parameter count, the same pretraining data, and the same inverse dynamics head would match or approach RynnWorld-4D-Policy's success rates, demonstrating that model scale and pretraining—not 4D geometric grounding—are the causal factors.","tokens_in":24033,"feed_emoji":"","tokens_out":1196,"duration_ms":127045,"temperature":0.7,"pith_summary":"The paper argues that jointly generating RGB, depth, and optical flow within a single diffusion process produces a 4D scene representation that is structurally closer to robot end-effector actions than 2D pixel video alone. The authors build RynnWorld-4D, a tri-branch video diffusion transformer that co-produces future RGB frames, depth maps, and optical flow from one RGB-D image and a language instruction, using cross-modal attention and 3D rotary position embeddings to keep appearance, geometry, and motion mutually consistent. They then attach an inverse dynamics head—RynnWorld-4D-Policy—that reads the frozen world model's internal 4D latents in a single forward pass and outputs 54-dimensional bimanual action chunks at roughly 9 Hz, bypassing multi-step denoising at inference. The central claim is that this predictive 4D representation space, which makes per-pixel 3D structure and inter-frame 3D scene flow explicit, narrows the gap between visual prediction and low-level control, yielding state-of-the-art success rates on six real-world dexterous bimanual tasks and most notably outperforming foundation policies like π0.5 on tasks demanding spatial precision and temporal coordination such as hand-over, lid placement, and bowl stacking.","feed_headline":"4D world model beats 2D policies on bimanual robot tasks","feed_subtitle":"Co-generating RGB, depth, and optical flow in one diffusion loop gives robot policies explicit 3D motion cues, narrowing the gap between预测视觉","key_machinery":"The RGB-DF tri-branch diffusion transformer with Joint Cross-Modal Attention and 3D RoPE, trained in three stages (independent modality adaptation, frozen-backbone joint attention, full fine-tuning) on 254.4 million pseudo-annotated frames, serves as a frozen 4D visual encoder. A Flow Former compresses its concatenated tri-branch hidden states into spatio-temporal tokens, and a 4-step flow-matching inverse dynamics head decodes 10-action chunks from these tokens.","core_discovery":"The paper's central discovery is that a frozen diffusion world model trained to jointly predict RGB, depth, and optical flow contains internal latent features that, when consumed directly by a lightweight inverse dynamics head in a single forward pass, outperform both standard 2D video policies and large vision-language-action foundation models on precision-critical bimanual manipulation. The mechanism is that depth and optical flow, when back-projected under pinhole-camera geometry, yield per-point 3D scene flow, making the world model's latent space carry explicit kinetic and geometric cues that 2D pixel latents must instead infer implicitly from appearance residuals.","pith_inferences":["The ablation comparing the full 4D encoder against a ResNet-18 baseline does not fully isolate whether the performance gain comes from 4D geometric grounding or from using a much larger pretrained transformer backbone. A cleaner test would compare the tri-branch 4D encoder against a single-branch transformer of equivalent parameter count and pretraining, which the paper does not provide.","The 9 Hz effective control frequency relies on action chunking (10 open-loop actions per 1.1 s planning cycle). If tasks require faster reactive correction within that 1.1 s window—such as catching a slipping object—the 9 Hz replan rate may be insufficient, and the paper's claim of robustness to mid-window perturbations is supported only qualitatively.","The pseudo-labels for depth and optical flow are produced by off-the-shelf foundation models (Depth Anything 3, DPFlow) trained on non-robotic data. Systematic biases in these pseudo-labels could propagate into the world model's 4D latents, and the paper does not measure sensitivity to pseudo-label quality.","If the core hypothesis—that explicit 3D scene flow latents are what drive the policy gains—holds, then adding a scene-flow consistency loss directly into the policy training objective (rather than relying on the frozen encoder to provide it implicitly) should further improve performance, which the paper does not test."],"forward_implications":["If predictive 4D latents are indeed closer to action space than 2D latents, future robot policies could standardize on 4D world-model encoders rather than 2D vision encoders, trading compute for geometric grounding.","The single-forward-pass extraction strategy suggests that expensive diffusion denoising is needed only during world-model training, not during closed-loop control, potentially resolving the latency bottleneck that has kept generative video models out of real-time robotics.","The 254M-frame Rynn4DDataset 1.0 with pseudo-labeled depth and flow could become a general training resource for other 4D embodied models beyond this specific architecture.","The tri-branch cross-modal attention design with shared K/V and per-branch FFNs may offer a template for fusing additional modalities (e.g., surface normals, tactile signals) into a single diffusion loop."],"fun_headline_variants":["4D world model gives robots explicit 3D motion cues for bimanual tasks","Frozen 4D diffusion model drives dexterous bimanual manipulation in one pass","4D latent features outperform 2D video policies on precision bimanual tasks","Joint RGB-depth-flow diffusion model improves spatial precision in robot manipulation"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The policy's performance gains are attributed to the 4D geometric and kinetic cues in the world model's latents, but the key ablation replaces the entire 4D encoder with a much smaller ResNet-18, making it impossible to distinguish whether the gains come from 4D structure or simply from using a vastly larger, pretrained transformer as the visual backbone.","fun_headline_variants_meta":{"raw":{"variants":["4D world model gives robots explicit 3D motion cues for bimanual tasks","Frozen 4D diffusion model drives dexterous bimanual manipulation in one pass","4D latent features outperform 2D video policies on precision bimanual tasks","Joint RGB-depth-flow diffusion model improves spatial precision in robot manipulation"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1156,"prompt_tokens":671,"completion_tokens":485,"prompt_tokens_details":null},"tokens_in":671,"tokens_out":485,"duration_ms":9623,"temperature":1.0,"reasoning_tokens":420,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T01:45:39.813804+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"A single-branch RGB-only transformer with the same parameter count, the same pretraining data, and the same inverse dynamics head would match or approach RynnWorld-4D-Policy's success rates, demonstrating that model scale and pretraining—not 4D geometric grounding—are the causal factors.","supporting_citations":[],"review_version":1}