{"id":"3e30c318-0b84-4191-bbb5-cfa69b639e15","arxiv_id":"2412.14484","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A fine-tuned Llama 3 LLM writes discrete human pose tokens from a text prompt, and a pose-conditioned diffusion video renderer turns them into videos, improving human motion fidelity.","lead":"DirectorLLM is a three-stage video generation system in which a fine-tuned Llama 3 model plans human poses from a text prompt and a separate renderer draws the video frames using those poses as a guide. The paper reports better human motion fidelity and prompt alignment than three existing text-to-video models, though the numerical margins are small and the evaluation set is narrowly filtered.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol is the load-bearing concern: the filtered 150-prompt set and VideoScore averages without error bars do not establish the claimed superiority over VideoCrafter2, with Visual Quality gains as small as 0.003.","rationale":"The reader identified the evaluation protocol as the weakest assumption, and I concur. The paper's central claim of superiority is a comparative statement, and the evidence for it is a 150-prompt filtered subset with no error bars. The appendix's explicit admission that gains are minimal on stationary prompts confirms that the reported results are conditional on excluding cases where the method does not help. The quantitative gaps over VideoCrafter2 are small, especially on Visual Quality (0.0034), which is likely within the noise of a learned metric. The human evaluation, described only as a figure without numerical results, cannot be independently checked. These are not fatal flaws—the architecture is plausible, the component ablations in Table 2 show internal consistency, and the qualitative examples are supportive—but they mean the performance claim is not yet established. My concrete test would settle whether the claim holds by using the full prompt distribution and explicit significance testing. Since my concern aligns with the reader's and does not change the conditional verdict, I recommend UNCHANGED.","tokens_in":13872,"tokens_out":4173,"duration_ms":36005,"concrete_test":"Re-evaluate DirectorLLM, VideoCrafter2, LaVie, and CogVideo on the complete unfiltered set of human-centric test prompts, including stationary and low-motion prompts, using VideoScore. Compute per-prompt scores and report paired bootstrap 95% confidence intervals and a paired permutation test between DirectorLLM and VideoCrafter2 on each metric. If the CLIPvid and dynamic-degree differences remain statistically significant (p < 0.05) while Visual Quality and Temporal Consistency do not, the claim should be scoped to motion-related metrics. If all differences become non-significant, the superiority claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DirectorLLM outperforms existing methods rests primarily on Table 1. The test set is 150 prompts filtered to dynamic-motion keywords, and the appendix admits 'We notice minimal performance gain with stationary prompts.' This is a post-hoc selection that favors the method. The largest gain over the strongest baseline (VideoCrafter2) is 0.047 on CLIPvid, but Visual Quality improves by only 0.0034, and no confidence intervals, significance tests, or per-prompt breakdowns are provided. VideoScore is a learned metric, so such small differences are plausibly within noise; a 0.003 difference on a 2.5-scale metric is far below any meaningful effect size. The human evaluation is reported only as a figure (Fig. 6) without percentages, counts, or inter-rater agreement, despite the appendix claiming over 5K ratings. Because every headline claim of superiority hinges on this evaluation, the lack of statistical rigor and the prompt filtering make the central claim unsupported. This is a correctness risk rather than an internal inconsistency, but it is load-bearing: without a rigorously unbiased evaluation, the paper's primary contribution cannot be confirmed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DirectorLLM, a three-stage system for human-centric text-to-video generation: (1) a Llama 3 8B LLM fine-tuned via next-token prediction to produce VQ-VAE-encoded pose tokens (20 keyframes at 1 FPS) from a text prompt; (2) a compact linear diffusion model that interpolates the sparse poses to 30 FPS, conditioned on CLIP text embeddings; and (3) a VideoCrafter2-based diffusion renderer augmented with a pose ControlNet. The authors evaluate on 150 dynamic human-centric prompts with VideoScore and a human A/B study, and ablate each component. They claim that this decoupling of motion planning from rendering yields improved human motion fidelity, prompt faithfulness, and subject naturalness over VideoCrafter2, LaVie, and CogVideo.","tokens_in":13928,"tokens_out":4990,"duration_ms":35761,"significance":"The modular design is a plausible and timely contribution: moving motion planning into an LLM token space is a clean way to inject explicit pose structure into text-to-video generation, and the component ablations in Table 2 (residual VQ, pose diffusion, text conditioning) show each stage contributes to the reported metrics. The approach is also renderer-agnostic in principle. However, the central claim of superiority over existing methods is not yet established because the quantitative evaluation uses a post-hoc-filtered prompt set and reports no uncertainty, while the human evaluation is summarized only as a figure. If the authors supply a less biased evaluation and proper statistics, the paper could be a useful contribution to LLM-guided video generation.","major_comments":[{"comment":"The headline comparison is evaluated on a test set of 150 prompts that were explicitly filtered to contain dynamic-motion keywords ('walking', 'running', 'yoga', etc.), and Appendix A3 acknowledges that 'We notice minimal performance gain with stationary prompts.' This post-hoc selection makes the reported superiority conditional on the subset where the method is expected to help and does not support the unqualified conclusion in Section 5 that the model 'outperforms existing methods.' Please report results on an unfiltered or balanced random sample of human-centric prompts, or explicitly restrict the claim to dynamic-motion scenarios and prespecify the filtering criterion.","section":"Section 3.3 and Appendix A3, Table 1"},{"comment":"The reported differences over the strongest baseline, VideoCrafter2, are extremely small on several dimensions (e.g., VC 2.5205 vs 2.5171, TC 2.4578 vs 2.4452), yet no variance, confidence intervals, or significance tests are reported. Since VideoScore is a learned metric, differences of this magnitude are plausibly within noise. Please provide per-prompt score distributions, standard errors, and a paired significance test (e.g., Wilcoxon signed-rank) for each metric, and interpret the effect sizes accordingly.","section":"Table 1"},{"comment":"The human A/B evaluation is presented only as a bar chart without raw preference percentages, counts per comparison, or statistical testing, despite Appendix D stating that over 5K ratings were collected. Please report the number of comparisons and raters per condition, the exact preference percentages, a binomial test or confidence intervals for each of the five questions, and inter-rater agreement for the majority-vote procedure.","section":"Section 3.3, Fig. 6, and Appendix D"},{"comment":"The evaluation prompts are randomly sampled from the same Shutterstock caption distribution used to train the DirectorLLM, whereas the baselines have not seen this distribution. This gives the proposed method a distributional advantage and weakens the fairness of the comparison. For a more neutral assessment, evaluate on prompts drawn from a different source or from standard video-caption benchmarks, keeping the human-centric and dynamic filtering identical across methods.","section":"Sections 3.1 and A3"}],"minor_comments":[{"comment":"The text says 'We show the results in Fig. 1 for key evaluation dimensions', but the quantitative results are in Table 1; Fig. 1 is the system overview figure.","section":"Section 3.3"},{"comment":"Section 3.3 lists 'VideoCrafter' among baselines, while Table 1 uses 'VideoCrafter2'; please make the naming consistent and specify the exact model version.","section":"Section 3.3, Table 1"},{"comment":"There are several typos that should be corrected: 'dimentional' to 'dimensional', 'represening' to 'representing', 'Pytorch-Lightining' to 'PyTorch Lightning', and '1−5' to '1e-5'.","section":"Appendix A2"},{"comment":"The text refers to 'Fig. 8' for the structure of the DirectorLLM and the interpolator, but those figures appear only in the appendix; please renumber or add cross-references to the main-text figures (Figs. 2 and 3).","section":"Section 2.2"},{"comment":"Appendix A1 states that poses were extracted for 25K videos but does not state how many videos remained after the human-centric and single-person filtering; please report the final training set size.","section":"Appendix A1"}],"recommendation":"major_revision","confidential_remarks":"The main barrier is the evaluation protocol. The appendix's own admission that stationary prompts show minimal gain, combined with tiny VideoScore differences and a bar-chart-only human study, makes the central claim currently unsupported. This is fixable within the paper's scope by adding an unfiltered evaluation, significance testing, and full human-study statistics, so I recommend major revision rather than rejection. The proprietary Shutterstock data may also limit reproducibility, but that alone would not be disqualifying for a systems paper of this type."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me cut to the chase. The actual novelty here is real: DirectorLLM is, as far as I know, the first system that fine-tunes an LLM (Llama 3) to emit residual VQ-VAE discretized pose tokens from text, then densifies and renders them. That is a sensible division of labor, and the three-stage pipeline is coherent. The component ablations in Table 2 actually do work: removing residual VQ coding drops Temporal Consistency from 2.457 to 1.815, and removing pose diffusion hurts too. Those internal comparisons give me confidence the system is doing something.\n\nWhat doesn't hold up is the headline comparison. The 150-prompt test set is filtered to dynamic-motion keywords, and the appendix admits \"We notice minimal performance gain with stationary prompts.\" That is post-hoc selection. The gains over VideoCrafter2 in Table 1 are mostly under 0.02, and Visual Quality improves by 0.0034 on a learned metric with no confidence intervals or significance tests. That is not a meaningful effect size. The human evaluation appears only as a preference figure without percentages or inter-rater agreement, even though the appendix claims over 5K ratings. The paper also skips direct comparison to the LLM-directed baselines it cites (VideoDirectorGPT, MotionGPT, Free-Bloom, LVD), so we cannot isolate whether the trained LLM beats a frozen planner.\n\nNone of this kills the architecture. The idea of offloading motion simulation to an LLM is plausible, and the ablations demonstrate each component contributes. What a referee should demand is: (1) evaluation on an unfiltered or random prompt set, or at least the full 150 before filtering; (2) error bars or bootstrap intervals on VideoScore; (3) at least one LLM-directed baseline; (4) numeric human evaluation results. Given the current evidence, I'd treat the system contribution as solid but the performance claim as unproven.\n\nThis is a paper worth engaging with. The audience is researchers working on controllable human-centric video generation and LLM-driven planning. It deserves peer review, but with the expectation of a major revision focused on evaluation rigor. I'd cite it for the architecture, not for the superiority claim.","headline":"Novel architecture worth a look, but the evaluation is too weak to support the headline claim of superiority over existing T2V models.","tokens_in":14718,"tokens_out":2023,"would_cite":true,"duration_ms":15791,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training an LLM to output human-pose tokens as a video director yields more faithful human motion than end-to-end text-to-video models.","keywords":["text-to-video generation","human motion planning","large language model director","pose token prediction","residual VQ-VAE","diffusion pose interpolation","ControlNet video generation","human-centric video generation"],"falsifier":"Run the same comparison on the full unfiltered prompt set, or on repeated seeds of the 150 dynamic prompts with confidence intervals, and check whether the VideoScore gaps and human-preference rates over VideoCrafter2 remain outside the noise; if they shrink to zero on stationary prompts or on resampled dynamic prompts, the paper's superiority claim fails.","tokens_in":13502,"feed_emoji":"🎬","tokens_out":8877,"duration_ms":66854,"temperature":0.7,"pith_summary":"The paper claims that text-to-video generation for people improves when the motion planning is handed to a trained large language model: the LLM reads the prompt and emits a discrete token sequence describing human poses at 1 frame per second, a lightweight diffusion model interpolates those poses to smooth 30 fps motion, and a pose-conditioned video generator renders the final frames. The central proposal is to take human-motion simulation out of the video diffusion model entirely, so the renderer can focus on drawing anatomically plausible frames instead of also deciding how bodies move. If the claim is right, prompts about running, dancing, yoga, and other dynamic actions will produce videos with fewer broken or extra limbs and better alignment to the requested activity than end-to-end text-to-video baselines. The paper supports the claim with automatic scores and blind A/B human judgments on 150 deliberately motion-heavy prompts, and with ablations showing that each of the three stages contributes to the final score.","feed_headline":"LLM as director sharpens human motion in generated video","feed_subtitle":"A tokenized pose plan from a language model, smoothed by diffusion, beats end-to-end models on dynamic human actions.","key_machinery":"The load-bearing device is the tokenized pose channel: a residual VQ-VAE with six codebooks of 512 tokens converts 18-keypoint 2D human skeletons, sampled at 1 fps into 20 key frames, into 120 discrete tokens, turning human motion into a language-model prediction problem. The fine-tuned LLM autoregressively predicts those tokens from the prompt, effectively acting as a layout and motion planner that also outputs instance-level bounding boxes. A linear diffusion interpolator, built on a prior pose-diffusion architecture, densifies the decoded sparse poses to 200 frames and conditions on CLIP text embeddings; finally a ControlNet layer with zero convolutions guides a frozen UNet-based video generator. The residual structure of the VQ-VAE is what preserves pose fidelity across the discrete bottleneck, and the ablation shows that losing it degrades visual quality more than losing the other two components.","core_discovery":"DirectorLLM is a three-stage architecture in which a Llama 3-based LLM is fine-tuned with next-token prediction to take the subject-level sentence of a caption and produce 120 vector-quantized pose tokens, representing 20 key-frame skeletons at 1 fps. The tokens are decoded back into poses, a linear diffusion model densifies them into 200 frames at 30 fps under CLIP text conditioning, and a ControlNet-augmented VideoCrafter2 renders the video from the dense pose sequence and the full caption. The paper's central claim is that this separation of labor lets the LLM absorb scene understanding and motion reasoning, so the system beats VideoCrafter, LaVie, and CogVideo on human motion fidelity, prompt faithfulness, and subject naturalness, especially for complex dynamics. The paper presents this as the first trained-LLM-as-video-director design, and its ablations show that removing the residual VQ-VAE hurts visual quality most, followed by removing pose diffusion and then by removing text conditioning in the interpolator.","pith_inferences":["The same director pattern could generalize beyond humans: any subject with a learnable discrete motion vocabulary (animals, vehicles, rigid bodies) could be planned by an LLM and rendered by a pose- or trajectory-conditioned generator, making the approach a general motion-planning interface.","Because the pose plan is text-generated and fully explicit, a natural next step the paper does not test is interactive editing: a user could revise the prompt and have the LLM re-emit only the changed pose tokens, giving fine-grained control over a single action without re-rendering the whole video.","The very small visual-quality gap over VideoCrafter2 (about 0.003 on VideoScore) suggests the honest advantage is in motion realism and prompt alignment, not pixel quality; a targeted evaluation measuring joint-angle plausibility or per-action success rates would show where the method actually wins."],"forward_implications":["Human-centric text-to-video becomes modular: because the DirectorLLM outputs ordinary pose frames, the same motion-planning module can be attached to different renderers (UNet or DiT) with minimal extra training.","The pose plan extends the effective video length: the LLM plans 200 frames of motion while the base renderer generates 16 frames at once, with FIFO-Diffusion stitching the clip together into a 7-second video.","The ablations imply that each stage is load-bearing: residual quantization preserves pose fidelity, the linear diffusion interpolator supplies smoothness, and text conditioning in the interpolator keeps motion aligned with the prompt.","The advertised gains are concentrated on dynamic actions (running, walking, dancing, yoga, jogging, jumping, cycling); the paper notes minimal performance gain on stationary prompts, so the method is a targeted improvement for motion-heavy human scenes."],"supporting_citations":[{"why":"Supplies the Llama 3 8B checkpoint and extended tokenizer from which DirectorLLM is fine-tuned.","marker":"Dubey et al. (2024)"},{"why":"Provides VideoCrafter, the frozen text-to-video UNet that the pose ControlNet augments.","marker":"Chen et al. (2023)"},{"why":"Supplies the ControlNet design with zero convolutions used to condition rendering on dense poses.","marker":"Zhang et al. (2023)"},{"why":"Provides the VQ-VAE discrete representation that turns continuous pose vectors into LLM-compatible tokens.","marker":"Van Den Oord et al. (2017)"},{"why":"OpenPose is used to extract the 18-keypoint human poses that make up the training signal and conditioning.","marker":"Cao et al. (2017)"},{"why":"VideoScore is the automatic evaluation metric behind the Table 1 quantitative comparisons.","marker":"He et al. (2024b)"},{"why":"Audio2PhotoReal is the architecture the linear pose interpolation diffusion model is built on.","marker":"Ng et al. (2024)"},{"why":"FIFO-Diffusion is used to extend the 16-frame renderer output to the full 200-frame pose sequence.","marker":"Kim et al. (2024)"},{"why":"Movie Gen supplies the same-person dataset filtering rule and the human A/B evaluation setup.","marker":"Polyak et al. (2024)"}],"fun_headline_variants":["Llama LLM directs human poses for better video generation","LLM as director plans poses, boosting video realism","DirectorLLM: language model choreographs human motion in video","Llama learns to direct poses for human-centric video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method is judged on 150 prompts filtered to motion-heavy keywords, and the reported automatic-score margins over the strongest baseline are tiny, so the whole superiority claim rests on those small margins being real signal rather than noise, especially since the paper itself notes minimal gain on stationary prompts.","fun_headline_variants_meta":{"raw":{"variants":["Llama LLM directs human poses for better video generation","LLM as director plans poses, boosting video realism","DirectorLLM: language model choreographs human motion in video","Llama learns to direct poses for human-centric video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1238,"prompt_tokens":964,"completion_tokens":274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":208}},"tokens_in":580,"tokens_out":274,"duration_ms":2193,"temperature":1.0,"reasoning_tokens":208,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:11:38.313347+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same comparison on the full unfiltered prompt set, or on repeated seeds of the 150 dynamic prompts with confidence intervals, and check whether the VideoScore gaps and human-preference rates over VideoCrafter2 remain outside the noise; if they shrink to zero on stationary prompts or on resampled dynamic prompts, the paper's superiority claim fails.","supporting_citations":[],"review_version":1}