VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System

Boxue Yang; Chunmei Qing; Dongzhi Jiang; Feng Zhao; Haodong Li; Haoyu Wu; Jialv Zou; Jinjing Zhao; Ji Xie; Juanxi Tian

VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Executable Blender code as a process-level chain of thought enforces physical consistency in text-to-video generation.

desk verdict Clever code-as-CoT video pipeline with solid benchmark gains, but the central decomposition claim is not actually proven by the ablation, and missing code/data makes results unverifiable. read the letter →

arxiv 2607.27380 v2 pith:7IKT5FY5 submitted 2026-07-29 cs.CV cs.AI

Haodong Li , Tianfei Ren , Xiaoxiao Ma , Chunmei Qing , Zhen Fang , Sipeng He , Ziyu Guo , Haoyu Wu

show 20 more authors

Juanxi Tian Yihang Zou Ruichuan An Dongzhi Jiang Boxue Yang Ji Xie Xu Huang Wenhao Yan Jialv Zou Zhengrong Yue Yaxin Luo Xiaotong Li Yuzhu Wang Junyan Ye Jinjing Zhao Zehui Chen Lin Chen Renye Yan Feng Zhao Pheng-Ann Heng

This is my paper · ORCID

classification cs.CVcs.AI

keywords videogenerationphysicalconsistencychain-of-thoughtexecutablecodeBlendersimulationdraft-conditionededitingtext-to-videoworldmodeling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that physically consistent video generation can be achieved by separating the physics from the pixels: have an agent write an executable Blender program specifying the scene and its temporal evolution, run it in a sandbox to get a deterministic low-fidelity draft, then condition a video editor on that draft to add photorealistic appearance. The claim is that this executable, frame-dense intermediate—a 'process-level chain of thought'—carries the physical dynamics, so the generative model only needs to restyle, not reason about how objects move. The authors report that this raises the average physical-consistency score from 0.475 to 0.558 on PhyGenBench and from 52.18% to 77.88% on VBench-2.0, the best average on both benchmarks, with the largest gains in thermal and material dynamics where appearance-driven priors are weakest. A sympathetic reader would care because it offers a controllable, inspectable way to inject physics into video generation without hand-authored rules per phenomenon.

What carries the argument

The central object is the 'process-level chain of thought': an executable Blender program that explicitly declares objects, physical properties, and temporal evolution, rendered into a deterministic spatiotemporal draft. Named in the paper as Code-as-CoT, this representation is explicit (every object and motion is declared), executable (the program actually runs and produces every frame), and inspectable (the code can be read, edited, and re-run). Its work in the argument is to commit the system to a concrete physical realization before any pixel is synthesized, so that the downstream generative video engine is asked only to restyle an already-instantiated process; the VideoCoCo-3K dataset s

What would settle it

Run the same PhyGenBench and VBench-2.0 prompt suites through VideoCoCo and the base generator and have human annotators with physics training rate which video better matches the requested dynamics; if human ratings do not reproduce a substantial gap favoring VideoCoCo, the central claim that the draft carries the physics is not established.

Watch

Extended reading notes

Core claim

The central claim is that executable code, not textual plans or keyframes, should be the intermediate reasoning representation for video generation. VideoCoCo externalizes the process hidden in a text prompt as a self-contained Blender Python program; running that program in a sandbox yields a deterministic, temporally dense draft of the event. A second engine, a video editor fine-tuned on the VideoCoCo-3K dataset of draft–instruction–target triplets, takes this draft as a structural condition and produces a photorealistic video. The paper argues that this decomposition lets the simulation engine commit to what happens and when, while the editor decides only how it looks, and presents benchm

Load-bearing premise

The load-bearing premise is that the LLM judges used by the benchmarks score genuine physical consistency, so the reported gains really reflect better physics rather than a bias toward the edited videos' appearance; the paper offers no human comparison to confirm this.

Editorial extensions

If this is right

If the claim holds, any prompt whose physics can be expressed in a simulator can be made physically consistent by drafting first, independent of the video model's own physical priors.
The largest gains in thermal and material dynamics suggest that appearance-driven generators are weakest exactly where an explicit simulator adds the most, pointing to a division of labor for future systems.
VideoCoCo-3K-style aligned triplets become a general training resource: any video editor can be adapted to consume simulated drafts, so improvements in the editor transfer to the whole pipeline.
Because the intermediate artifacts are code and rendered video, failures can be debugged by editing the program, not by re-prompting the generator; this makes physical errors inspectable and correctable.
LoRA adaptation outperforming full fine-tuning indicates that the needed skill—restyling drafts while preserving motion—is narrow and can be learned with modest data, lowering the cost of adapting the pipeline to new editors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same code-as-draft idea could be pushed one step further: instead of a video editor conditioned on a draft, a video model could be trained with synthetic intermediate targets so that at inference the simulation is not needed, as the authors themselves suggest for future work—but the paper does not demonstrate this.
Because the staged construction is automatic, a natural extension is to close the loop with a verifier that checks whether the rendered draft matches the prompt before editing; the paper does not implement such a rejection loop.
The benchmark gains are only as trustworthy as the judges; a human rating study on the same prompt set would test whether the gap reflects genuine physical plausibility rather than a stylistic preference for the editor's outputs.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a genuinely new way to inject physical priors into video generation — write Blender code, render a deterministic draft, and condition a video editor on it. That's worth taking seriously. But the paper claims 'drafting supplies most of the physical gain' in a way that its own Table 3 does not support: Tune-Free adds 0.03 over OmniWeaving, LoRA tuning adds another 0.05. So editor adaptation contributes a larger chunk of the score, and without a direct measure of whether final videos preserve the draft's trajectories, the central decomposition is more asserted than established.

What's good: the framework is clean and well-motivated, the idea of executable code as a process-level chain of thought is a real extension beyond DraCo/CoCo for images, and the ablation structure (tuning-free vs LoRA vs full) is the right way to try to isolate the draft contribution. The benchmark results, if trustworthy, are strong — best average on PhyGenBench and VBench-2.0 physical dimensions, with particularly large gains on material and thermal. The paper also honestly notes its own limits (latency, Blender expressiveness).

Where the softness is: first, the ablation interpretation. The text says the draft supplies most of the physical gain, but the numbers show the opposite in terms of raw improvement. You could defend it as 'the draft enabled the rest,' but that's not what's written. Second, both benchmarks depend on LLM judges (GPT-4o for PhyGenBench; an undefined 'pre-registered subset' for VBench-2.0) with no human correlation reported. That's a field-wide issue, but here it's load-bearing because the headline gains are large and the judge could be rewarding restyled appearance over physical fidelity. Third, there's no check — automatic or human — that the final video preserves the simulated spatiotemporal structure. That's the core claim, and it's unmeasured. Fourth, no code/data/checkpoints are provided, so none of this is independently checkable, and the dataset is self-built with the same pipeline components, including a Seedance teacher, adding a real but not fatal circularity.

This paper is for people working on physics-aware video generation who want an alternative to pure reward-based post-training. It deserves a serious referee, but that referee should ask for the missing artifacts and an honest re-analysis of the ablation before the strong claims are accepted.

Referee Report

4 major / 5 minor

Summary. The paper proposes VideoCoCo, an agentic dual-engine framework for physically consistent text-to-video generation. A coding agent writes a Blender program from a text prompt; the program is executed in a sandbox to render a deterministic, low-fidelity 'white-clay' spatiotemporal draft. An instruction agent composes an appearance-focused editing instruction, and a video editor is conditioned jointly on the draft and instruction to produce the final photorealistic video. To make off-the-shelf editors consume simulated drafts, the authors construct VideoCoCo-3K, a dataset of draft–instruction–target triplets generated by a teacher editor (Seedance 2.0), and fine-tune the editor with a conditional denoising objective (Eq. 7). Experiments on PhyGenBench and VBench-2.0 report substantial gains over the OmniWeaving base (0.475 to 0.558 and 52.18% to 77.88%, respectively), with ablations separating tuning-free and fine-tuned variants. The central claim is that executable code serves as a process-level chain of thought, decoupling physical dynamics from appearance synthesis.

Significance. If the claims hold, VideoCoCo is a meaningful step toward controllable and inspectable physical video generation: unlike textual plans or keyframes, executable code can be run, inspected, and revised, and the dense simulated draft provides explicit temporal grounding. The paper includes a useful ablation isolating the draft's contribution from editor adaptation, reports per-category scores on two physics benchmarks, and identifies clear failure cases (e.g., turbulent fluids). However, the strength of the central decomposition claim depends on untested assumptions about the LLM judges used for evaluation and on the attribution of gains in Table 3, which currently appears to contradict the text. The absence of released code, data, and evaluation details further limits verification. The idea is promising and the empirical framework is coherent, but the evidence as presented does not yet fully support the paper's strongest claims.

major comments (4)

[§4.3, Table 3; §4.2 'Takeaway'] The text states that 'Drafting supplies most of the physical gain,' but the Table 3 numbers show the opposite attribution. OmniWeaving→Tune-Free improves the average by 0.506−0.475 = 0.031, while Tune-Free→LoRA-Tune improves by 0.558−0.506 = 0.052; thus editor adaptation contributes roughly 63% of the total gain. This is a load-bearing inconsistency for the central decomposition claim. Please correct the claim, or provide a controlled edit-only baseline (e.g., omitting the draft but keeping the instruction and LoRA adaptation) that would isolate the draft's contribution.
[§4.1 'Datasets and Metrics'] PhyGenBench scores rely entirely on GPT-4o as the MLLM judge, and VBench-2.0 uses a 'pre-registered subset' that is never defined. No human agreement, judge discriminability analysis, or complete benchmark results are reported. Since the editor restyles appearance (lighting, texture, camera), the observed gains could reflect judge preference for cleaner restyled videos rather than physical fidelity. Please report judge–human correlation (or at least a human-rated subset), specify the exact VBench-2.0 prompts/suite used, and justify why the subset is representative.
[§3.4, Eq. (7)] The training setup is underspecified. Eq. (7) is a generic conditional denoising loss, but the architecture changes needed to condition the editor on the draft video d are not described: is d injected as concatenated frames, via cross-attention, or through an adapter? What LoRA rank, learning rate, training steps, and compute were used? Without these details the 'LoRA-Tune outperforms Full-Tune' result cannot be reproduced or interpreted. Additionally, the VideoCoCo-3K targets are generated by Seedance 2.0 using the same draft-and-instruction pipeline; there is no verification that the teacher outputs actually preserve the draft's spatiotemporal trajectories, which is critical because the editor learns to imitate the teacher.
[Reproducibility] The project page (https://github.com/micky-li-hd/VideoCoCo) is listed but no code, data, or checkpoints are provided in the manuscript. Given the paper's reliance on a new dataset and a tuned editor, and its claim of reproducibility ('fully automatic, inspectable, and reproducible'), releasing at least the dataset metadata, Blender programs, and evaluation scripts is necessary for the claims to be verifiable. Please clarify the release plan.

minor comments (5)

[Table 1 vs. §4.2] The abstract and text use 0.475/0.558 while Table 1 rounds to 0.48/0.56; please standardize to avoid apparent inconsistency.
[Figure 2] The figure contains placeholder symbols ('??') in the draft rendering chain, which make the pipeline unclear. Use actual rendered draft frames or remove the placeholders.
[§2.3] 'VideoRL VR' reads as a typo; clarify the method name and citation.
[§4.1] The term 'pre-registered subset' appears only here; define it in the text or in an appendix, including how it was selected from VBench-2.0.
[§3.4, Eq. (7)] The loss is written as plain L2 on the noise residual; specify whether any reweighting (e.g., SNR weighting) is used, and whether the model is trained with classifier-free guidance on the draft condition.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Code-as-CoT claim rests on external benchmarks and a tuning-free ablation; the self-citations and self-built dataset do not make the prediction equivalent to its inputs.

full rationale

The derivation chain is empirical: p -> c = A_code(p) (Eq. 1), d = B(c) (Eq. 2), e = A_edit(p,d) (Eq. 3), v = G_theta(d,e) (Eq. 4). The only fitted component, G_theta, is adapted on VideoCoCo-3K triplets whose targets are produced by an external teacher Seedance 2.0 (Eq. 5-6), and Sec. 3.3 excludes evaluation-benchmark prompts and near-duplicates, so benchmark scores are not obtained by feeding the training labels back through the model. The central claim that the draft supplies dynamics is isolated by the tuning-free ablation in Sec. 4.3 (0.475 -> 0.506 with no editor parameter update), which is an empirical comparison rather than a construction; no equation in the paper reduces the predicted physical consistency to the fitted objective by definition. The Code-as-CoT framing cites the authors' own CoCo/DraCo papers, but those self-citations are motivational, not load-bearing evidence, and the LoRA gain is normal teacher distillation from an external model rather than a parameter renamed as a prediction. Two concerns are noted for the correctness pass, not the circularity pass: the Sec. 4.3 takeaway "Drafting supplies most of the physical gain" overstates the table's deltas (LoRA adaptation, 0.506 -> 0.558, contributes more than the draft, 0.475 -> 0.506), and the LLM-judge validity on PhyGenBench/VBench-2.0 is not checked against human ratings. Section 5's stated limitations (latency, Blender expressiveness) similarly affect scope, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 2 invented entities

The paper's central claim does not rest on fitted numerical constants; the main burden is on unstated evaluation details, judge reliability, and the self-generated training set. No explicit free parameters are disclosed, which is itself a gap because dataset size/quality and judge configuration are implicit choices.

assumptions (4)

domain assumption The sandboxed Blender execution is deterministic and the rendered draft faithfully captures the physical process intended by the prompt.
The method relies on the code agent writing correct-enough Blender programs, and on the draft video preserving the target dynamics. The paper acknowledges simulator expressiveness limits (Sec. 5), but treats the drafts as trustworthy scaffolds (Sec. 3.1).
domain assumption LLM judge scores (GPT-4o for PhyGenBench; VBench-2.0 protocol) are a valid proxy for physical consistency.
The headline results are completely mediated by these judges; no human correlation or error analysis is provided (Sec. 4.1, Tables 1-2).
domain assumption OmniWeaving is a suitable base editor, and its weights/training setup can be adapted with Eq. 7 as described.
The model is initialized from OmniWeaving and the paper omits architectural specifics of how the draft conditions the editor; this assumption is load-bearing for the reported gains.
domain assumption VideoCoCo-3K triplets, generated by Seedance, provide supervision that transfers to new prompts.
The dataset is created by the same pipeline (Sec. 3.3) and evaluation prompts are excluded, but no analysis of distribution/shift or contamination is given beyond the exclusion claim.

invented entities (2)

VideoCoCo-3K dataset
purpose: Training data aligning Blender drafts with photorealistic targets to adapt the video editor.
The dataset has no external validation; it is constructed with the authors' own pipeline and is not released in the manuscript.
Draft-conditioned editor adaptation paradigm (Code-as-CoT for video) independent evidence
purpose: Intermediate representation for physically consistent video generation.
The generated videos and benchmark scores provide falsifiable handles outside the paper, but no artifact release is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System." pith.science (2026). https://pith.science/paper/7IKT5FY5

@misc{pith2026260727380,
  author       = {Pith},
  title        = {Pith review of: VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7IKT5FY5}},
  note         = {Machine review of arXiv:2607.27380}
}

read the original abstract

Text-to-video models have achieved remarkable visual quality, yet they still struggle to generate physically consistent dynamics because the temporal evolution of a scene must be inferred implicitly from a highly compressed text prompt. Existing chain-of-thought approaches introduce intermediate plans or visual states, but these representations are typically non-executable or temporally sparse, limiting their ability to instantiate and control the complete spatiotemporal process. To address this limitation, we introduce VideoCoCo, an agentic dual-engine framework in which executable Blender code serves as a process-level chain of thought. Given a text prompt, a coding agent synthesizes a Blender program that explicitly specifies the scene and its temporal evolution. The executable simulation engine runs the program to produce a deterministic spatiotemporal draft, which is subsequently transformed into a photorealistic video by a generative video engine through draft-conditioned editing. This decomposition separates process-level reasoning from high-fidelity visual realization. To adapt the video editor to simulated drafts, we construct VideoCoCo-3K, a curated dataset of draft-instruction-target triplets. VideoCoCo improves the OmniWeaving baseline from 0.475 to 0.558 on PhyGenBench and from 52.18 to 77.88 on VBench-2.0, achieving the best average score on both benchmarks. These results demonstrate that executable code provides an effective, controllable, and inspectable intermediate representation for physically consistent video generation.

Discussion (0). Continue with ORCID to comment.

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.