{"id":"5d5b421e-0b6f-4874-b4b7-0898936fdd05","arxiv_id":"2412.00596","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM-guided, iterative prompt-refinement system that captions generated videos, detects physics and semantic mismatches, and rewrites prompts, improving physics-adherence scores on VideoPhy and PhyGenBench.","lead":"This paper presents PhyT2V, a prompt-refinement system that uses an LLM to iteratively rewrite text-to-video prompts after checking what the generated video actually shows. On VideoPhy and PhyGenBench, it reports up to 2.3x higher physics-adherence scores than the unrefined model, but the same score is also used as feedback inside the loop.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2.3x/35% gains are not independently established because the loop optimizes the same VideoCon-Physics metric used for evaluation; a blinded human or held-out metric check is needed.","rationale":"The reader's weakest assumption—that the video captioning model must be faithful—is valid and under-tested, but I see a more fundamental issue: the feedback signal and the outcome measure are the same instrument. In §3.2 the score [S] is part of the Step-3 prompt, in §3.3 convergence is judged by that score, and in §4 the reported PC/SA numbers come from the identical VideoCon-Physics evaluator. This makes the 2.3x/35% claim a measure of the method's ability to optimize its own objective, not an independent demonstration of physical realism. The paper does include ablations and failure cases, which are good signs, and the method may well be effective; what is missing is an external yardstick. A blinded human rating or a pre-registered held-out metric not used in the loop would settle the question. Since the reader's CONDITIONAL verdict already captures the need for independent evaluation, my read does not change the verdict, hence UNCHANGED, with partial agreement because the reader emphasized caption fidelity rather than metric circularity.","tokens_in":26195,"tokens_out":5053,"duration_ms":51556,"concrete_test":"Run a blinded human evaluation on a random 100-prompt subset of VideoPhy using CogVideoX-5B, having three annotators independently rate the physical plausibility of Round-1 versus Round-4 videos without knowing which is which; if human-rated physical plausibility does not reproduce a large fraction of the reported PC gain, the headline improvement is a metric artifact rather than evidence of better physical grounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—up to 2.3x improvement in physical-rule adherence—is measured exclusively by the VideoCon-Physics evaluator. That same evaluator's score [S] is injected into Step 3 of the refinement loop (§3.2) and is used as the stopping criterion (§3.3). PhyT2V is therefore explicitly optimizing the evaluation metric: the LLM sees the score, can discover the metric's blind spots, and the reported PC/SA gains may reflect better exploitation of VideoCon-Physics rather than better real-world physics. The paper provides no human evaluation, no independent physics metric, and no evidence that VideoCon-Physics is accurate on the specific generated videos. A secondary but related gap is that Step 2's mismatch signal depends on Tarsier's caption fidelity, which is also unmeasured; however, even with perfect captions the shared-metric problem would remain. This is not an internal inconsistency, but it means the central quantitative claim is not yet empirically secured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PhyT2V, a training-free, data-independent method that refines text-to-video prompts through an LLM-driven loop. In each round, Step 1 asks the LLM to identify the main object and relevant physical rules; Step 2 uses a video captioning model (Tarsier) to summarize the generated video and an LLM to find semantic mismatches between the caption and the user prompt; Step 3 uses step-back reasoning, together with a VideoCon-Physics score [S] of the previous round's video, to produce a refined prompt. The loop continues until the VideoCon-Physics score is satisfactory or improvement converges. The authors apply PhyT2V to CogVideoX-5B, CogVideoX-2B, OpenSora, and VideoCrafter on the VideoPhy and PhyGenBench benchmarks, reporting up to 2.3x improvement in physical-rule adherence over the unrefined model and at least 35% improvement over two prompt-enhancer baselines. The central claim is that prompt-only, LLM-guided iterative refinement can push existing T2V models toward more physically plausible videos without retraining or extra inputs.","tokens_in":26351,"tokens_out":4601,"duration_ms":51392,"significance":"If the reported gains hold up under independent measurement, PhyT2V would be a useful and low-cost contribution: it is model-agnostic, requires no retraining, and its three-step prompt templates are clearly specified and easy to reproduce. The paper also includes extensive qualitative comparisons, ablations of both reasoning steps, an honest appendix of failure cases, and a public code link. The main weakness is that the headline metric, VideoCon-Physics PC/SA, is also the feedback signal used inside the refinement loop, so the measured gains may reflect optimization of the evaluator rather than genuine physical adherence. The paper would be substantially strengthened by a human evaluation or a held-out metric not used in the loop, plus caption-fidelity checks, and by extending the baseline comparison to all four T2V models.","major_comments":[{"comment":"The headline PC/SA gains in Tables 1 and 2 are measured by the VideoCon-Physics evaluator, and the same evaluator's score [S] is injected into the Step 3 prompt (§3.2) and used as the stopping criterion (§3.3). The refinement loop is therefore explicitly optimizing the metric on which the paper's central claim rests: the LLM sees whether the previous prompt scored above or below 0.5, and the process terminates when the same evaluator reports convergence. This does not make the method internally inconsistent, but it means the reported 2.3x and 35% improvements are not independent evidence of improved physical adherence. I ask the authors to (i) report PC/SA with a held-out evaluator that was not used in the loop, (ii) run a blinded human evaluation on a stratified sample of generated videos, and (iii) report the agreement between VideoCon-Physics and human judgments on the generated videos. In addition, Tables 1–6 report only averages over binary outputs, with no confidence intervals or significance tests, so the reliability of the Round 2–4 increments is not established.","section":"§3.2, §3.3, §4"},{"comment":"The mismatch signal in Step 2 depends on Tarsier's captions being faithful summaries of the generated videos. If the caption hallucinates content or omits a physical violation, the refined prompt will correct a problem that does not exist in the video or will leave the real problem uncorrected. The paper does not measure caption accuracy on the evaluated videos, and Appendix D's failure cases show that the T2V models can diverge substantially from prompts. Please add a caption-fidelity check, such as human evaluation of a random sample of captions or a caption-video consistency score, and report how often the Step 2 mismatch was judged correct on that sample.","section":"§3.2, Step 2"},{"comment":"The claim of 'at least 35% improvement' over prompt enhancers is supported only for CogVideoX-5B and OpenSora; CogVideoX-2B and VideoCrafter are absent from Tables 3 and 4. Since the paper claims PhyT2V is generic across all four tested models, the baseline comparison should be extended to the remaining two models, or the claim should be restricted to the evaluated subset.","section":"§4.1, Tables 3 and 4"},{"comment":"The out-of-distribution generality claim, stated in the abstract and introduction, is not quantified. No OOD split, distribution-shift measure, or separate OOD metric is provided; Figure 3 shows only selected anecdotal examples. I recommend either defining an OOD evaluation split and reporting the relevant metrics separately, or softening the claim to reflect that out-of-distribution behavior is demonstrated qualitatively.","section":"§1, §4.1, Figure 3"}],"minor_comments":[{"comment":"The phrase 'Physcial-ground Text-to-Video' contains a typo and should read 'Physics-grounded Text-to-Video'.","section":"§3, opening paragraph"},{"comment":"'PhyT2V is high generic' should be 'PhyT2V is highly generic'.","section":"§4, bullet list"},{"comment":"The phrase 'which throne momentarily in the ambient light' appears to contain a typo; 'throne' should probably be 'hover' or 'hang'.","section":"Figure 11 caption"},{"comment":"The VBench results are presented only as a radar-style figure without numeric values; please report the numerical scores in a table or appendix so the improvements are independently readable.","section":"Figure 7 and surrounding text"},{"comment":"The sentence 'The diffusion model generated video length is setted as 6 second and 8 frames per second' should be revised to 'set to 6 seconds and 8 frames per second'.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is the main risk to the paper's quantitative claims. If the authors add a human evaluation or a held-out metric and extend the baselines to all four models, the contribution would be solid; I do not see a novelty disclosure problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about PhyT2V. First, it is a real, new combination: an inference-time loop that captions the generated video, has an LLM compare the caption to the prompt, and rewrites the prompt with chain-of-thought and step-back reasoning, iterating until a score threshold. That is distinct from the cited static prompt enhancers and layout planners. Second, the headline 2.3x/35% gains are not independently established, because the same VideoCon-Physics evaluator is both the refinement signal in Step 3 and the outcome metric. That is a genuine circularity, not a quibble.\n\nWhat the paper does well: it is clearly written, the three-step prompt templates are concrete, the ablations show both reasoning steps matter, and the authors tested on four T2V models and two benchmarks, including VBench. They also document failure cases in Appendix D, which is more honest than most papers in this space. The method is data-independent and works with frozen models, which is practically useful if it holds up.\n\nThe soft spots are the usual ones, but they land harder because of the metric loop. No error bars or significance tests anywhere; baselines compared on only two of the four models; and the captioning model Tarsier is trusted without validation on these specific videos. A wrong caption means the LLM fixes a mismatch that does not exist. The VBench results are a partial mitigation: they are a held-out metric not used in the loop, and they do improve. But the abstract's 2.3x and the 35% comparisons come from the same VideoCon-Physics score used as feedback. Also minor: the abstract says 2.3x while Table 1 shows 2.2x for PC on CogVideoX-2B; and the out-of-distribution claim is not quantified.\n\nWho this is for: people working on prompt engineering, LLM-guided generation, or T2V evaluation. It is not a breakthrough paper, but it is a useful increment with a clear evaluation flaw. My recommendation: send it to peer review. The method deserves referee time, and the reviewers can push for an independent human study or a held-out physics metric, plus variance reporting. That is exactly the kind of revision that would make the claims credible.","headline":"Genuinely new inference-time prompt refinement loop for physics-grounded T2V, but the headline numbers rest on a circular metric; treat the 2.3x/35% as unverified until an independent evaluation lands.","tokens_in":26914,"tokens_out":2115,"would_cite":false,"duration_ms":20085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PhyT2V claims that an iterative LLM refinement loop, which reads a caption of the current video and rewrites the prompt to close semantic and physical gaps, makes existing text-to-video models obey real-world physical rules up to 2.3x…","keywords":["text-to-video generation","diffusion transformers","chain-of-thought prompting","step-back prompting","physical commonsense","prompt refinement","video captioning feedback","LLM-guided generation"],"falsifier":"Take the evaluated prompts, generate videos with known physical errors (for example, an apple that splits into multiple bouncing pieces), caption those videos with Tarsier, and test whether the captions mention the violation; if captions are frequently silent or wrong, the mismatch step either corrects a non-existent problem or leaves the real one untouched. A direct rerun with a different video captioning model would also settle whether the reported gains come from the refinement loop or from idiosyncrasies of Tarsier's descriptions.","tokens_in":25960,"feed_emoji":"🎬","tokens_out":8678,"duration_ms":69719,"temperature":0.7,"pith_summary":"The paper tries to establish that physical violations in AI-generated video can be reduced without retraining the generator or feeding it extra conditions such as depth maps or 3D geometry. The method, PhyT2V, runs an iterative loop in which an LLM first names the physical rules the scene should obey, a video captioning model converts the current video into text, the LLM identifies the mismatch between that caption and the user's prompt, and the LLM then rewrites the prompt to close the gap. The claim is that this text-only feedback loop improves adherence to real-world physical rules by up to 2.3x on existing text-to-video models and outperforms prompt enhancers by at least 35%. If true, the result would make text prompts a sufficient control surface for physical realism, offering a general, data-independent alternative to training on larger physics datasets.","feed_headline":"LLM prompt rewriting makes AI videos obey physics up to 2.3x better","feed_subtitle":"A caption-feedback loop fixes physical errors in text-to-video models without retraining or extra inputs.","key_machinery":"The central mechanism is the iterative self-refinement loop, which decomposes prompt rewriting into two parallel subproblems and one synthesis subproblem. Local chain-of-thought reasoning is used inside Step 1 (physical rules) and Step 2 (mismatch detection), while global step-back reasoning is used in Step 3 to combine those abstractions into a refined prompt; the video captioning model is the load-bearing bridge that keeps all reasoning in text. An external physics evaluator (VideoCon-Physics, a 0-to-1 score for physical commonsense and semantic adherence) supplies the feedback that decides whether another round is needed.","core_discovery":"On the paper's own terms, the discovery is that chain-of-thought and step-back reasoning can be applied to text-to-video generation entirely in the text domain, using a caption of the generated video as the bridge. In each refinement round, Step 1 extracts the main object and the physical laws it should obey, Step 2 compares the video caption against the user prompt to find semantic and physical mismatches, and Step 3 produces a refined prompt that resolves the mismatch under the stated physical laws. The paper reports that this loop raises physical-commonsense and semantic-adherence scores across CogVideoX-5B, CogVideoX-2B, OpenSora, and VideoCrafter on the VideoPhy and PhyGenBench benchmarks, with gains up to 2.3x over the unrefined model and at least 35% over ChatGPT-4 and Promptist prompt enhancers, typically converging within three to four rounds.","pith_inferences":["A natural extension, not tested in the paper, is to use the same generate-caption-compare-rewrite loop for other generative modalities whose outputs can be verbalized by a captioner or describer, such as text-to-image or text-to-3D.","The paper's stated bound is the captioning model's fidelity; a fair comparison would measure Tarsier's caption accuracy on the evaluated videos, since a hallucinated or over-literal caption would send the loop after a problem that is not in the video.","A testable follow-up is to replace the binary mismatch description with the evaluator's continuous score as the refinement signal, and check whether the loop converges faster or reaches higher final adherence.","Because the method requires no retraining and no model-internal access, it could plausibly be applied to closed, API-only T2V systems, which the paper does not demonstrate."],"forward_implications":["Existing T2V models should regain physical plausibility on out-of-distribution prompts through pure prompt rewriting, with most of the gain appearing in the first two refinement rounds.","The method transfers across model families with different architectures and input formats, since it only changes the text the model receives.","Prompt enhancers that do not inspect the generated video are expected to lag behind PhyT2V by at least 35% on physical-commonsense benchmarks.","The refinement loop needs only a few rounds before improvements converge, so the added cost is a handful of extra generations plus LLM calls per prompt.","VBench video-condition-consistency dimensions should improve alongside the physical-commonsense metrics, as reported in the paper's VBench results."],"supporting_citations":[{"why":"supplies the VideoPhy prompts and the VideoCon-Physics evaluator used to measure physical commonsense and semantic adherence and to decide when to stop refining.","marker":"[2]"},{"why":"supplies the PhyGenBench prompts across mechanics, optics, and thermal domains that are used for out-of-distribution evaluation.","marker":"[29]"},{"why":"is the CogVideoX T2V model family that motivates the method and receives the largest reported improvements.","marker":"[51]"},{"why":"is the VideoCrafter model used to demonstrate that the loop generalizes to another T2V architecture.","marker":"[4]"},{"why":"is the OpenSora model used to demonstrate generality and to compare against prompt-enhancer baselines.","marker":"[53]"},{"why":"is the Tarsier video captioning model that converts each generated video into text for mismatch detection.","marker":"[39]"},{"why":"is the ChatGPT o1-preview LLM that performs the chain-of-thought and step-back reasoning in all three steps.","marker":"[18]"},{"why":"is the Promptist baseline whose prompt enhancement PhyT2V must outperform.","marker":"[21]"},{"why":"serves as the ChatGPT-4 prompt-enhancer baseline that PhyT2V is compared against.","marker":"[28]"}],"fun_headline_variants":["PhyT2V: LLM self-refinement makes AI videos obey physics 2.3x better","No retraining, just prompt rewrites: LLM loop boosts physics in T2V","Caption feedback loop lets LLMs make T2V physics-compliant","Self-refining prompts improve physics adherence in T2V by 2.3x","LLM-guided iterative refinement teaches T2V models physics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the video captioning model describes the generated video faithfully and completely enough for the LLM to see the actual physical violation; the paper does not measure caption accuracy on the evaluated videos.","fun_headline_variants_meta":{"raw":{"variants":["PhyT2V: LLM self-refinement makes AI videos obey physics 2.3x better","No retraining, just prompt rewrites: LLM loop boosts physics in T2V","Caption feedback loop lets LLMs make T2V physics-compliant","Self-refining prompts improve physics adherence in T2V by 2.3x","LLM-guided iterative refinement teaches T2V models physics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000715,"raw_usage":{"total_tokens":3203,"prompt_tokens":919,"completion_tokens":2284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":2176}},"tokens_in":535,"tokens_out":2284,"duration_ms":44599,"temperature":1.0,"reasoning_tokens":2176,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:11:50.044641+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the evaluated prompts, generate videos with known physical errors (for example, an apple that splits into multiple bouncing pieces), caption those videos with Tarsier, and test whether the captions mention the violation; if captions are frequently silent or wrong, the mismatch step either corrects a non-existent problem or leaves the real one untouched. A direct rerun with a different video captioning model would also settle whether the reported gains come from the refinement loop or from idiosyncrasies of Tarsier's descriptions.","supporting_citations":[{"cited_title":"Open-sora: Democratizing efficient video production for all, 2024","cited_arxiv_id":null,"evidence_quote":"is the OpenSora model used to demonstrate generality and to compare against prompt-enhancer baselines."},{"cited_title":"Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models","cited_arxiv_id":null,"evidence_quote":"is the VideoCrafter model used to demonstrate that the loop generalizes to another T2V architecture."},{"cited_title":"Introducing openai o1-preview, 2024","cited_arxiv_id":null,"evidence_quote":"is the ChatGPT o1-preview LLM that performs the chain-of-thought and step-back reasoning in all three steps."},{"cited_title":"Promptist: Auto- mated prompt optimization for text-to-image synthesis","cited_arxiv_id":null,"evidence_quote":"is the Promptist baseline whose prompt enhancement PhyT2V must outperform."},{"cited_title":"Gpt4motion: Scripting physical motions in text-to-video generation via blender-oriented gpt planning,","cited_arxiv_id":null,"evidence_quote":"serves as the ChatGPT-4 prompt-enhancer baseline that PhyT2V is compared against."}],"review_version":1}