Pith. sign in

REVIEW 2 major objections 3 minor 60 cited by

VideoPhy: Evaluating Physical Commonsense for Video Generation

T0 review · 2 major / 3 minor · reviewed 2026-05-20 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Text-to-video models generate videos that follow both captions and physical laws in fewer than 40 percent of cases.

desk verdict VideoPhy gives a useful new benchmark showing top video models hit only 39.6% on physical commonsense in human tests, but the lack of agreement metrics leaves the headline number a bit soft. read the letter →

arxiv 2406.03520 v2 pith:QZHGDGY5 submitted 2024-06-05 cs.CV cs.AIcs.LG

Hritik Bansal , Zongyu Lin , Tianyi Xie , Zeshun Zong , Michal Yarom , Yonatan Bitton , Chenfanfu Jiang , Yizhou Sun
show 2 more authors
Kai-Wei Chang Aditya Grover
This is my paper
classification cs.CVcs.AIcs.LG
keywords text-to-videogenerationphysicalcommonsensebenchmarkvideoevaluationgenerativemodelshuman
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces VideoPhy, a benchmark of text prompts that describe everyday physical interactions between solids, fluids, and other materials. Researchers generate videos from multiple current models and ask human evaluators to check whether each video matches the prompt and also obeys real-world physics such as rolling, pouring, or breaking. Even the strongest model tested, CogVideoX-5B, succeeds on both criteria only 39.6 percent of the time. The results indicate that present generators are still far from functioning as accurate simulators of the physical world. The authors also release an automated scorer, VideoCon-Physics, to evaluate future models more quickly.

What carries the argument

The VideoPhy benchmark, which supplies diverse prompts involving material-type interactions and measures success via human judgment of caption adherence plus physical-law compliance.

What would settle it

A new model that produces videos judged by humans to follow both the prompt and physical laws on more than 70 percent of VideoPhy instances would weaken the claim that current generators lack physical commonsense.

Watch

Extended reading notes

Core claim

VideoPhy reveals that existing text-to-video generative models severely lack the ability to generate videos adhering to the given text prompts while also lacking physical commonsense, with the best model succeeding on only 39.6 percent of instances.

Load-bearing premise

Human evaluators can reliably and consistently judge whether a generated video follows physical commonsense for the curated prompts.

Editorial extensions

If this is right

  • Video generative models remain far from accurately simulating the physical world.
  • Progress on future models can be tracked with the released VideoPhy prompts and protocol.
  • The automated VideoCon-Physics evaluator can be applied to newly released models without repeated human studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Improved performance on VideoPhy could make generated videos more usable for planning tasks that require realistic motion.
  • Weak results on fluid-solid interactions may indicate specific gaps that targeted training data or loss terms could address.
  • The same curation approach could be extended to create benchmarks for other forms of commonsense such as object permanence or causal chains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper introduces VideoPhy, a benchmark for assessing physical commonsense in text-to-video generative models. It curates prompts involving material interactions (solid-solid, solid-fluid, fluid-fluid), generates videos from open and closed SOTA models (e.g., CogVideoX-5B, Lumiere), and reports human evaluation results showing that even the best model adheres to both the caption and physical laws in only 39.6% of cases. The work also proposes an automatic evaluator, VideoCon-Physics, for scalable assessment of future models.

Significance. If the human evaluation results hold, this benchmark provides concrete evidence of a substantial gap in current video generation models' ability to simulate real-world physics, which is important for their potential use as general-purpose simulators. The direct use of human judgments on curated physical interactions supplies falsifiable, model-agnostic evidence rather than relying on self-referential metrics. The proposal of VideoCon-Physics is a constructive addition for reproducibility and future work. The evaluation across both open and closed models and the focus on diverse material-type interactions are particular strengths.

major comments (2)
  1. [Human Evaluation] Human evaluation protocol: the central 39.6% figure for CogVideoX-5B (and all other reported percentages) is presented without inter-annotator agreement statistics (e.g., Fleiss' kappa or pairwise agreement) or error bars on the physical-laws label. Because the claim that models 'severely lack' physical commonsense rests directly on these human judgments, the absence of agreement data makes it difficult to separate model failure from annotator variance.
  2. [Benchmark Construction] Prompt curation and validation: the description of how prompts were selected and verified to test genuine physical commonsense (rather than ambiguous or underspecified cases) remains high-level. More detail on the curation process, including any expert review or pilot testing for physical accuracy, would be needed to establish that the benchmark instances are load-bearing tests of the claimed capability gap.
minor comments (3)
  1. [Abstract] The abstract and results sections use the phrase 'severely lack' for the 39.6% figure; a more precise statement of the quantitative gap would improve tone and clarity.
  2. [Results] Table or figure presenting per-category breakdown (solid-solid vs. fluid-fluid, etc.) would help readers assess whether failures are uniform or concentrated in particular interaction types.
  3. [Auto-Evaluator] The auto-evaluator VideoCon-Physics is introduced but its correlation with human judgments and any ablation on its training data are not detailed enough for independent reproduction.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the constructive feedback on our paper. We address the major comments below and plan to incorporate revisions to improve the clarity and rigor of our human evaluation and benchmark construction sections.

read point-by-point responses

  1. Referee: [Human Evaluation] Human evaluation protocol: the central 39.6% figure for CogVideoX-5B (and all other reported percentages) is presented without inter-annotator agreement statistics (e.g., Fleiss' kappa or pairwise agreement) or error bars on the physical-laws label. Because the claim that models 'severely lack' physical commonsense rests directly on these human judgments, the absence of agreement data makes it difficult to separate model failure from annotator variance.

    Authors: We agree that reporting inter-annotator agreement is important for validating the reliability of our human evaluation results. In the revised manuscript, we will include Fleiss' kappa scores for the annotations on physical adherence and caption adherence. Additionally, we will provide error bars or confidence intervals for the reported percentages to better quantify the variability in the human judgments. This will help demonstrate that the observed low performance is indeed due to model limitations rather than annotator disagreement. revision: yes

  2. Referee: [Benchmark Construction] Prompt curation and validation: the description of how prompts were selected and verified to test genuine physical commonsense (rather than ambiguous or underspecified cases) remains high-level. More detail on the curation process, including any expert review or pilot testing for physical accuracy, would be needed to establish that the benchmark instances are load-bearing tests of the claimed capability gap.

    Authors: We thank the referee for this suggestion. In the original manuscript, we provided a high-level overview of the prompt curation to maintain focus on the evaluation results. However, we acknowledge that additional details would enhance the reproducibility and credibility of the benchmark. In the revised version, we will expand the section on benchmark construction to include more specifics on the prompt selection criteria, the process of verifying physical accuracy through pilot studies, and any expert consultations or reviews conducted to ensure the prompts test genuine physical commonsense without ambiguity. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity in benchmark evaluation or auto-evaluator proposal

full rationale

The paper curates a set of text prompts involving physical interactions across material types and evaluates outputs from existing text-to-video models via human judgment on adherence to both captions and physical laws. No equations, parameter fitting, or first-principles derivations are claimed; the 39.6% figure for CogVideoX-5B is a direct empirical count from external model generations and annotator labels. The proposed VideoCon-Physics auto-evaluator is introduced as a new tool without reducing to any self-citation chain or redefinition of inputs. All load-bearing steps rely on independent human evaluation protocols and publicly available generative models rather than internal consistency loops.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on the assumption that the curated prompts adequately sample physical commonsense and that human raters apply consistent criteria; no free parameters are fitted, no new entities are postulated, and axioms are standard assumptions about physical laws and evaluation validity.

assumptions (1)
  • domain assumption Human raters can accurately detect violations of physical commonsense in short video clips.
    Invoked in the human evaluation section to interpret the 39.6% adherence rate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VideoPhy: Evaluating Physical Commonsense for Video Generation." pith.science (2026). https://pith.science/paper/QZHGDGY5

@misc{pith2026240603520,
  author       = {Pith},
  title        = {Pith review of: VideoPhy: Evaluating Physical Commonsense for Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QZHGDGY5}},
  note         = {Machine review of arXiv:2406.03520}
}
read the original abstract

Recent advances in internet-scale video data pretraining have led to the development of text-to-video generative models that can create high-quality videos across a broad range of visual concepts, synthesize realistic motions and render complex objects. Hence, these generative models have the potential to become general-purpose simulators of the physical world. However, it is unclear how far we are from this goal with the existing text-to-video generative models. To this end, we present VideoPhy, a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities (e.g. marbles will roll down when placed on a slanted surface). Specifically, we curate diverse prompts that involve interactions between various material types in the physical world (e.g., solid-solid, solid-fluid, fluid-fluid). We then generate videos conditioned on these captions from diverse state-of-the-art text-to-video generative models, including open models (e.g., CogVideoX) and closed models (e.g., Lumiere, Dream Machine). Our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts, while also lack physical commonsense. Specifically, the best performing model, CogVideoX-5B, generates videos that adhere to the caption and physical laws for 39.6% of the instances. VideoPhy thus highlights that the video generative models are far from accurately simulating the physical world. Finally, we propose an auto-evaluator, VideoCon-Physics, to assess the performance reliably for the newly released models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Apple-$\pi$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

    cs.CV 2026-07 conditional novelty 7.0 of 10

    A law-grounded benchmark, Apple-PI, grades video models stage-by-stage on physics reasoning and finds they top out at 0.473, well short of reliable simulation.

  2. Multiplayer Interactive World Models with Representation Autoencoders

    cs.CV 2026-07 accept novelty 7.0 of 10

    A 5B-parameter latent diffusion model generates real-time four-player Rocket League matches conditioned on all players' actions, staying stable far beyond its training horizon.

  3. ScratchWorld: Evaluating If World Models Compute Executable Consequences

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    ScratchWorld benchmark finds that language models achieve at most 13.8% value-aware changed-field F1 on replay-verified Scratch state transitions and frequently ignore executable rules.

  4. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    MemoBench is a new diagnostic benchmark with 360 synthetic and real clips plus VQA evaluation that tests memory consistency in video models under the disappear-and-reappear paradigm in dynamically changing environments.

  5. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    MemoBench curates 360 ground-truth clips and an evaluation suite to diagnose memory consistency failures in video models when objects change state while out of view.

  6. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 conditional novelty 7.0 of 10

    None of ten tested video-generation models reliably remembers objects after occlusion in dynamic scenes; static-camera videos inflate consistency scores.

  7. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 conditional novelty 7.0 of 10

    Current video world models do not reliably recover an object's updated state after it disappears and reappears under simultaneous camera and scene dynamics.

  8. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    MemoBench is a new diagnostic benchmark with automated and VQA metrics that evaluates memory consistency in video models under disappear-and-reappear in dynamic environments.

  9. Physics in 2-Steps: Locking Motion Priors Before Visual Refinement Erases Them

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    PhaseLock extracts motion priors from 2-step inference and enforces them via Latent Delta Guidance to raise physical consistency scores by 6.2 points on average in image-to-video diffusion models.

  10. Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Dream.exe evaluates 8 video generation models on 101 manipulation tasks by converting generated videos into executable robot trajectories in a simulator, finding measurable success rates that visual metrics do not predict.

  11. MBench: A Comprehensive Benchmark on Memory Capability for Video World Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    MBench is a new benchmark that quantifies long-term memory in video world models via three hierarchical consistency dimensions evaluated on curated real videos.

  12. YoCausal: How Far is Video Generation from World Model? A Causality Perspective

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    YoCausal benchmark shows video diffusion models detect the arrow of time but lack genuine causal understanding relative to humans.

  13. What-If World: A Causal Benchmark for General World Models in Embodied Scenarios

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    What-If World is a new paired-prompt benchmark showing that nine state-of-the-art video generation models achieve at most 52% on causal intervention tests and cluster near 28% for open-source systems.

  14. WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    WBench is a benchmark with 289 test cases and 1,058 turns for evaluating interactive world models using 22 automated metrics validated against human judgments.

  15. CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    CRONOS benchmark shows recent open-source video generators fail to preserve physical consistency under controlled changes to viewpoint, scene, object category, and appearance.

  16. MechVerse: Evaluating Physical Motion Consistency in Video Generation Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    MechVerse benchmark shows current video generation models preserve appearance but fail at mechanically admissible motion, with errors rising as coupling complexity increases.

  17. PhyGround: Benchmarking Physical Reasoning in Generative World Models

    cs.CV 2026-05 accept novelty 7.0 of 10

    PhyGround is a new benchmark with curated prompts, a 13-law taxonomy, large-scale human annotations, and an open physics-specialized VLM judge for evaluating physical reasoning in generative video models.

  18. Do Joint Audio-Video Generation Models Understand Physics?

    cs.SD 2026-05 unverdicted novelty 7.0 of 10

    AV-Phys Bench shows that current joint audio-video models lack robust physical commonsense, with major drops on transitions and deliberate anti-physics prompts.

  19. Do Joint Audio-Video Generation Models Understand Physics?

    cs.SD 2026-05 unverdicted novelty 7.0 of 10

    Current joint audio-video generation models lack robust physical commonsense, especially during transitions and when prompted for impossible behaviors.

  20. CMTA: Leveraging Cross-Modal Temporal Artifacts for Generalizable AI-Generated Video Detection

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    CMTA detects AI-generated videos by capturing unnatural temporal stability in visual-textual semantic alignment via joint embeddings and multi-grained temporal modeling, outperforming prior methods in cross-generator tests.

  21. MoRight: Motion Control Done Right

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    MoRight disentangles object and camera motion via canonical-view specification and temporal cross-view attention, while decomposing motion into active user-driven and passive consequence components to learn and apply ...

  22. PlayWorld: Learning Robot World Models from Autonomous Play

    cs.RO 2026-03 unverdicted novelty 7.0 of 10

    PlayWorld learns high-fidelity robot world models from unsupervised self-play, producing physically consistent video predictions that outperform models trained on human data and enabling 65% better real-world policy p...

  23. VideoASMR-Bench: Can AI-Generated ASMR Videos Fool VLMs and Humans?

    cs.CV 2025-12 unverdicted novelty 7.0 of 10

    VideoASMR-Bench shows state-of-the-art VLMs fail to reliably detect AI-generated ASMR videos from real ones, though humans can still identify the fakes relatively easily.

  24. DreamGen: Unlocking Generalization in Robot Learning through Video World Models

    cs.RO 2025-05 unverdicted novelty 7.0 of 10

    DreamGen trains robot policies on synthetic trajectories from adapted video world models, enabling a humanoid robot to perform 22 new behaviors in seen and unseen environments from a single pick-and-place teleoperatio...

  25. A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Models

    cs.CV 2026-06 conditional novelty 6.5 of 10

    CrashTwin recovers metric-scale crash dynamics from monocular rollouts and shows that strong visual scores routinely mask large momentum, energy, and identity violations in world models.

  26. Learning Explicit Physical Parameter Control and Benchmarking for Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Explicit instance-level physical parameter conditioning with routing attention improves physical-law consistency in image-to-video generation, as measured on the authors' new simulator-based benchmark.

  27. SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation

    cs.AI 2026-07 unverdicted novelty 6.0 of 10

    SimWorlds presents a multi-agent system with planner-coder-reviewer workflow, layered scene protocol, and runtime inspection tools to create dynamic 4D scenes from text, plus the 4DBuildBench benchmark showing outperf...

  28. A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    CrashTwin is a new benchmark framework that exposes physical violations in state-of-the-art world models during multi-agent collisions despite high visual quality.

  29. A Good Talk Does not Look Like a Summary, It Teaches You! Measuring Takeaways from Paper-to-Video Talks

    cs.MM 2026-06 unverdicted novelty 6.0 of 10

    EffectivePresentationScorer evaluates paper-to-video talks for instructional quality by checking clear explanation of ideas, prerequisite concepts, and links to contributions, finding that current systems cover topics...

  30. MemoBench: Benchmarking World Modeling in Dynamically Changing Environments

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    MemoBench curates 360 clips and an evaluation suite to test video models on recovering updated object states after disappear-and-reappear in changing environments.

  31. GEOPHYS: The Geometry of Physical Plausibility

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    GEOPHYS defines five geometric properties of per-frame embeddings from image encoders that detect physical implausibility in videos with SOTA accuracy and serve as an efficient verifier.

  32. VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Introduces VideoWeaver benchmark (16 categories, 285 cases) plus agent-as-judge and skill-evolution algorithm to assess and improve agentic long video generation across frameworks.

  33. LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    LaMo adds self-supervised latent motion priors via a motion drift loss during training and motion prior guidance during sampling to boost physical fidelity in video diffusion models like CogVideoX.

  34. NEWTON: Agentic Planning for Physically Grounded Video Generation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    NEWTON improves physical accuracy in video generation by deploying a trainable planner that coordinates physics-aware tools and a verifier, raising joint accuracy on VideoPhy-2 without altering the base generators.

  35. Enhancing Train-Free Infinite-Frame Generation for Consistent Long Videos

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    MIGA introduces two-stage alignment to close train-inference gaps and dual consistency enhancement via self-reflection and long-range guidance to achieve SOTA temporal consistency in infinite-frame video generation on...

  36. PanoWorld: Geometry-Consistent Panoramic Video World Modeling

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    PanoWorld adds depth consistency and trajectory consistency losses plus spherical adaptations to a pre-trained video model, plus a new PanoGeo dataset, to produce geometry-consistent 360 video.

  37. Quantitative Video World Model Evaluation for Geometric-Consistency

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    PDI-Bench computes 3D projective residuals from segmented and tracked points to quantify geometric inconsistency in AI-generated videos.

  38. How Far Are Video Models from True Multimodal Reasoning?

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Current video models succeed on basic understanding but achieve under 25% success on logically grounded generation and near 0% on interactive generation, exposing gaps in multimodal reasoning.

  39. ATSS: Detecting AI-Generated Videos via Anomalous Temporal Self-Similarity

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    ATSS detects AI-generated videos by measuring unnatural repetitive temporal correlations in triple similarity matrices derived from frame visuals and semantic descriptions.

  40. Measuring 3D Spatial Geometric Consistency in Dynamic Video Generation

    cs.CV 2026-03 conditional novelty 6.0 of 10

    SGC quantifies 3D geometric consistency of generated videos by measuring divergence among local camera poses estimated only on static background sub-regions.

  41. IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video

    cs.CV 2026-03 accept novelty 6.0 of 10

    IRIS releases 220 real 4K videos of eight dynamical systems with ground-truth parameters plus a protocol that measures parameter recovery, equation selection, and multi-body failure modes of unsupervised video-to-phys...

  42. VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction

    cs.CV 2026-02 unverdicted novelty 6.0 of 10

    VisPhyWorld evaluates MLLMs' physical reasoning via executable code generation for video reconstruction, with VisPhyBench showing strong semantics but weak parameter inference and dynamics simulation.

  43. PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A new dataset and fine-tuned VLM detector/explainer called PhyDetEx shows that current T2V models still struggle to generate videos that obey physical laws, with open-source models performing worse.

  44. Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A learned human-action manifold, combining 3D pose, 2D keypoints, appearance, and motion derivatives, scores generated videos by distance to real-action centroids and embedding smoothness, beating prior metrics on hum...

  45. SURF: Signature-Retained Fast Video Generation

    cs.GR 2025-11 unverdicted novelty 6.0 of 10

    SURF accelerates high-resolution video generation up to 12.5x by using noise reshifting for low-res previews from pretrained models and a shifting-window Refiner for efficient upscaling that retains original signatures.

  46. PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding

    cs.CV 2025-11 conditional novelty 6.0 of 10

    A two-stage framework that predicts per-part material properties from a single image and uses editable physics simulation to guide video generation.

  47. RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scaling

    cs.CV 2025-10 unverdicted novelty 6.0 of 10

    RAPO++ is a three-stage prompt optimization framework combining retrieval-augmented refinement, closed-loop test-time scaling, and LLM fine-tuning to enhance text-to-video generation quality.

  48. AEGIS: Authenticity Evaluation Benchmark for AI-Generated Video Sequences

    cs.CV 2025-08 conditional novelty 6.0 of 10

    AEGIS is a large-scale benchmark for detecting AI-generated videos, with a hard test set of Sora and KLing clips that current vision-language models detect at near-chance accuracy.

  49. Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations

    cs.RO 2025-07 unverdicted novelty 6.0 of 10

    RIGVid shows that filtered AI-generated videos can serve as effective supervision for complex robotic manipulation tasks without any real demonstrations.

  50. MAGI-1: Autoregressive Video Generation at Scale

    cs.CV 2025-05 unverdicted novelty 6.0 of 10

    MAGI-1 is a 24B-parameter autoregressive video world model that predicts denoised frame chunks sequentially with increasing noise to enable causal, scalable, streaming generation up to 4M token contexts.

  51. Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation

    cs.CV 2024-10 unverdicted novelty 6.0 of 10

    PhyGenBench supplies 160 prompts across 27 physical laws and an automated LLM/VLM evaluation pipeline to measure physical commonsense compliance in current text-to-video models.

  52. LivePhys: Transforming Static Physics Problems into Interactive Simulations via a Scan-to-Play Framework

    cs.ET 2026-07 conditional novelty 5.0 of 10

    LivePhys, a three-stage pipeline combining image segmentation, OCR, and an MLLM with a physics engine, outperforms general-purpose multimodal models at generating executable, interactive Matter.js simulations from sta...

  53. Your Data Manifold is Secretly a Reward Model: Shell-LCC for Text-to-Video Generation

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    Shell-LCC models the high-quality data manifold as an isotropic shell to derive cost-free reward signals that improve realism and high-frequency details in text-to-video generation.

  54. Your Data Manifold is Secretly a Reward Model: Shell-LCC for Text-to-Video Generation

    cs.CV 2026-06 conditional novelty 5.0 of 10

    Generated video latents are pushed toward a shell-shaped manifold fitted to high-quality SFT video patches, producing a dense reward that reduces blur, over-smoothing, and motion artifacts in text-to-video models.

  55. Physics-IQ Verified

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    Physics-IQ Verified refines 57.6% of samples and 34.8% of prompts from the original benchmark and produces moderate ranking shifts (Kendall's τ = 0.46) across six image-to-video models.

  56. Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI

    cs.AI 2026-06 conditional novelty 5.0 of 10

    Kairos learns and maintains control-sufficient world states via a cross-embodiment curriculum, hybrid linear temporal attention, and deployment-aware co-design for Physical AI.

  57. Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI

    cs.AI 2026-06 unverdicted novelty 5.0 of 10

    Kairos is a native world model stack using cross-embodiment pretraining, hybrid linear temporal attention with theoretical error bounds, and deployment-aware co-design, reporting top performance on embodied benchmarks.

  58. CP4D: Compositional Physics-aware 4D Scene Generation

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    CP4D generates physically consistent 4D scenes via compositional integration of pre-trained 3D models, hybrid simulator-diffusion motion synthesis, and automated scene composition.

  59. Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    PILA aligns frozen flow-matching video models to a physics attribute bank via MoE experts and operational residuals, reporting SOTA physical plausibility on VBench-2.0, VideoPhy-2 and PhyGenBench while preserving visu...

  60. MPMWorlds: Material-Point-Method Simulations for Inferring and Extrapolating Physical Dynamics

    cs.GR 2026-06 unverdicted novelty 5.0 of 10

    Assembles MPM simulation dataset and compares code generation versus video diffusion for inferring physical parameters and extrapolating dynamics from videos.

Reference graph

Works this paper leans on

135 extracted references · 135 canonical work pages · cited by 58 Pith papers

  1. [1]

    Luma Dream Machine | AI Video Generator — lumalabs.ai

    Luma AI. Luma Dream Machine | AI Video Generator — lumalabs.ai. https://lumalabs. ai/dream-machine, 2024

    work page 2024

  2. [2]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021

    work page 2021

  3. [3]

    Videocon: Robust video-language alignment via contrast captions

    Hritik Bansal, Yonatan Bitton, Idan Szpektor, Kai-Wei Chang, and Aditya Grover. Videocon: Robust video-language alignment via contrast captions. arXiv preprint arXiv:2311.10111, 2023

    work page arXiv 2023

  4. [4]

    Talc: Time-aligned captions for multi-scene text-to-video generation

    Hritik Bansal, Yonatan Bitton, Michal Yarom, Idan Szpektor, Aditya Grover, and Kai-Wei Chang. Talc: Time-aligned captions for multi-scene text-to-video generation. arXiv preprint arXiv:2405.04682, 2024

    work page arXiv 2024

  5. [5]

    Comparing bad apples to good oranges: Aligning large language models via joint preference optimization

    Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, and Aditya Grover. Comparing bad apples to good oranges: Aligning large language models via joint preference optimization. arXiv preprint arXiv:2404.00530, 2024

    work page arXiv 2024

  6. [6]

    How well can text-to- image generative models understand ethical natural language interventions? arXiv preprint arXiv:2210.15230, 2022

    Hritik Bansal, Da Yin, Masoud Monajatipoor, and Kai-Wei Chang. How well can text-to- image generative models understand ethical natural language interventions? arXiv preprint arXiv:2210.15230, 2022

    work page arXiv 2022

  7. [7]

    Lumiere: A space-time diffusion model for video generation.arXiv preprint arXiv:2401.12945, 2024

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024

    work page arXiv 2024

  8. [8]

    An introduction to physically based modeling: rigid body simulation i—unconstrained rigid body dynamics

    David Baraff. An introduction to physically based modeling: rigid body simulation i—unconstrained rigid body dynamics. SIGGRAPH course notes, 82, 1997

    work page 1997

Show all 135 references
  1. [9]

    A fast variational framework for accurate solid-fluid coupling

    Christopher Batty, Florence Bertails, and Robert Bridson. A fast variational framework for accurate solid-fluid coupling. ACM Transactions on Graphics (TOG), 26(3):100–es, 2007

    2007

  2. [10]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  3. [11]

    Visit-bench: A benchmark for vision-language instruction following inspired by real-world use

    Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. arXiv preprint arXiv:2308.06595, 2023

    2023

  4. [13]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    2023 arXiv

  5. [14]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2023

  6. [15]

    Fluid simulation for computer graphics

    Robert Bridson. Fluid simulation for computer graphics. AK Peters/CRC Press, 2015

    2015

  7. [16]

    Generating long videos of dynamic scenes

    Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei Efros, and Tero Karras. Generating long videos of dynamic scenes. Advances in Neural Information Processing Systems, 35:31769–31781, 2022. 12

    2022

  8. [17]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024

    2024

  9. [18]

    Genie: Generative interactive environments

    Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. arXiv preprint arXiv:2402.15391, 2024

    2024

  10. [19]

    Storybench: A multifaceted benchmark for continuous story visualization

    Emanuele Bugliarello, H Hernan Moraldo, Ruben Villegas, Mohammad Babaeizadeh, Moham- mad Taghi Saffar, Han Zhang, Dumitru Erhan, Vittorio Ferrari, Pieter-Jan Kindermans, and Paul V oigtlaender. Storybench: A multifaceted benchmark for continuous story visualization. Advances i...

    2024

  11. [20]

    cerspense/zeroscope_v2_576w · Hugging Face — huggingface.co

    cerspense. cerspense/zeroscope_v2_576w · Hugging Face — huggingface.co. https:// huggingface.co/cerspense/zeroscope_v2_576w, 2023

    2023

  12. [21]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047, 2024

    2024

  13. [22]

    Physical simulation of environmentally induced thin shell deformation

    Hsiao-Yu Chen, Arnav Sastry, Wim M van Rees, and Etienne V ouga. Physical simulation of environmentally induced thin shell deformation. ACM Transactions on Graphics (TOG), 37(4):1–13, 2018

    2018

  14. [23]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:2402.19479, 2024

    2024

  15. [24]

    Multi-layer thick shells

    Yunuo Chen, Tianyi Xie, Cem Yuksel, Danny Kaufman, Yin Yang, Chenfanfu Jiang, and Minchen Li. Multi-layer thick shells. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–9, 2023

    2023

  16. [25]

    Learning universal policies via text-guided video generation

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  17. [26]

    A survey on machine learning approaches for modelling intuitive physics

    Jiafei Duan, Arijit Dasgupta, Jason Fischer, and Cheston Tan. A survey on machine learning approaches for modelling intuitive physics. arXiv preprint arXiv:2202.06481, 2022

    2022

  18. [27]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023

    2023

  19. [28]

    Scaling rectified flow transform- ers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In Forty-first International Conference on Machin...

    2024

  20. [29]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  21. [30]

    Iq-mpm: an interface quadrature material point method for non-sticky strongly two-way coupled nonlinear solids and fluids

    Yu Fang, Ziyin Qu, Minchen Li, Xinxin Zhang, Yixin Zhu, Mridul Aanjaneya, and Chenfanfu Jiang. Iq-mpm: an interface quadrature material point method for non-sticky strongly two-way coupled nonlinear solids and fluids. ACM Transactions on Graphics (TOG), 39(4):51–1, 2020

    2020

  22. [31]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Syste...

    2024

  23. [32]

    genmo. Genmo. Create videos and images with AI. — genmo.ai. https://www.genmo.ai/. 13

  24. [33]

    Maniskill2: A unified benchmark for generalizable manipulation skills

    Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskill2: A unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.04659, 2023

    2023

  25. [34]

    A convex formulation of frictional contact between rigid and deformable bodies

    Xuchen Han, Joseph Masterjohn, and Alejandro Castro. A convex formulation of frictional contact between rigid and deformable bodies. IEEE Robotics and Automation Letters, 2023

    2023

  26. [35]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  27. [36]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022

    2022 arXiv

  28. [37]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  29. [38]

    Dreamphysics: Learning physical properties of dynamic 3d gaussians with video diffusion priors

    Tianyu Huang, Yihan Zeng, Hui Li, Wangmeng Zuo, and Rynson WH Lau. Dreamphysics: Learning physical properties of dynamic 3d gaussians with video diffusion priors. arXiv preprint arXiv:2406.01476, 2024

    2024

  30. [39]

    Plasticinelab: A soft-body manipulation benchmark with differentiable physics

    Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. arXiv preprint arXiv:2104.03311, 2021

    2021

  31. [40]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. arXiv preprint arXiv:2311.17982, 2023

    2023

  32. [41]

    EulerDiscreteScheduler — huggingface.co

    huggingfaceEulerDiscreteScheduler. EulerDiscreteScheduler — huggingface.co. https: //huggingface.co/docs/diffusers/en/api/schedulers/euler

  33. [42]

    Text2video-zero: Text-to-image diffusion mod- els are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion mod- els are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Compute...

    2023

  34. [43]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  35. [44]

    Drucker-prager elastoplasticity for sand animation

    Gergely Klár, Theodore Gast, Andre Pradhana, Chuyuan Fu, Craig Schroeder, Chenfanfu Jiang, and Joseph Teran. Drucker-prager elastoplasticity for sand animation. ACM Transactions on Graphics (TOG), 35(4):1–12, 2016

    2016

  36. [45]

    KLING AI — klingai.com

    KlingAI. KLING AI — klingai.com. https://www.klingai.com/, 2024

    2024

  37. [46]

    Videopoet: A large language model for zero-shot video generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023

    2023 arXiv

  38. [47]

    Smoothed particle hydrodynamics techniques for the physics based simulation of fluids and solids

    Dan Koschier, Jan Bender, Barbara Solenthaler, and Matthias Teschner. Smoothed particle hydrodynamics techniques for the physics based simulation of fluids and solids. arXiv preprint arXiv:2009.06944, 2020

    2009

  39. [48]

    Subjective-aligned dateset and metric for text-to-video quality assessment

    Tengchuan Kou, Xiaohong Liu, Zicheng Zhang, Chunyi Li, Haoning Wu, Xiongkuo Min, Guangtao Zhai, and Ning Liu. Subjective-aligned dateset and metric for text-to-video quality assessment. arXiv preprint arXiv:2403.11956, 2024

    2024

  40. [49]

    Viescore: Towards explain- able metrics for conditional image synthesis evaluation

    Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explain- able metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312.14867, 2023. 14

    2023

  41. [50]

    GitHub - LAION-AI/aesthetic-predictor: A linear estimator on top of clip to predict the aesthetic quality of pictures — github.com

    LaionAI. GitHub - LAION-AI/aesthetic-predictor: A linear estimator on top of clip to predict the aesthetic quality of pictures — github.com. https://github.com/LAION-AI/ aesthetic-predictor, 2022

    2022

  42. [51]

    Variational stokes: a unified pressure- viscosity solver for accurate viscous liquids

    Egor Larionov, Christopher Batty, and Robert Bridson. Variational stokes: a unified pressure- viscosity solver for accurate viscous liquids. ACM Transactions on Graphics (TOG), 36(4):1– 11, 2017

    2017

  43. [52]

    Aligning text-to-image models using human feedback

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023

    2023 arXiv

  44. [53]

    User experience rating scales with 7, 11, or 101 points: does it matter? Journal of Usability Studies, 12(2), 2017

    James R Lewis and O˘guzhan Erdinç. User experience rating scales with 7, 11, or 101 points: does it matter? Journal of Usability Studies, 12(2), 2017

    2017

  45. [54]

    Incremental potential contact: intersection- and inversion-free, large-deformation dynamics

    Minchen Li, Zachary Ferguson, Teseo Schneider, Timothy R Langlois, Denis Zorin, Daniele Panozzo, Chenfanfu Jiang, and Danny M Kaufman. Incremental potential contact: intersection- and inversion-free, large-deformation dynamics. ACM Trans. Graph., 39(4):49, 2020

    2020

  46. [55]

    Codimensional incremental potential contact

    Minchen Li, Danny M Kaufman, and Chenfanfu Jiang. Codimensional incremental potential contact. arXiv preprint arXiv:2012.04457, 2020

    2012

  47. [56]

    Aligning diffusion models by optimizing human utility

    Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. arXiv preprint arXiv:2404.04465, 2024

    2024

  48. [57]

    Energetically consistent inelasticity for optimiza- tion time integration

    Xuan Li, Minchen Li, and Chenfanfu Jiang. Energetically consistent inelasticity for optimiza- tion time integration. ACM Transactions on Graphics (TOG), 41(4):1–16, 2022

    2022

  49. [58]

    Gpu-accelerated robotic simulation for distributed reinforcement learning

    Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, and Dieter Fox. Gpu-accelerated robotic simulation for distributed reinforcement learning. In Conference on Robot Learning, pages 270–282. PMLR, 2018

    2018

  50. [59]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.01291, 2024

    2024

  51. [60]

    Physics3d: Learning physical properties of 3d gaussians via video diffusion

    Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, and Yueqi Duan. Physics3d: Learning physical properties of 3d gaussians via video diffusion. arXiv preprint arXiv:2406.04338, 2024

    2024

  52. [61]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  53. [62]

    Physgen: Rigid-body physics-grounded image-to-video generation

    Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics-grounded image-to-video generation

  54. [63]

    Evalcrafter: Benchmarking and evaluating large video generation models

    Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  55. [64]

    Sora: A review on background, technology, limitations, and opportunities of large vision models

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024

    2024 arXiv

  56. [65]

    Effect of the number of response categories on the reliability and validity of rating scales

    Luis M Lozano, Eduardo García-Cueto, and José Muñiz. Effect of the number of response categories on the reliability and validity of rating scales. Methodology, 4(2):73–79, 2008

    2008

  57. [66]

    Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022. 15

    2022 arXiv

  58. [67]

    Physically- aware generative network for 3d shape modeling

    Mariem Mezghanni, Malika Boulkenafed, Andre Lieutier, and Maks Ovsjanikov. Physically- aware generative network for 3d shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9330–9341, 2021

    2021

  59. [68]

    mplug-owl-video

    mplugowl. mplug-owl-video. https://github.com/X-PLUG/mPLUG-Owl/tree/main/ mPLUG-Owl/mplug_owl_video

  60. [69]

    Particle-based fluid-fluid interaction

    Matthias Müller, Barbara Solenthaler, Richard Keiser, and Markus Gross. Particle-based fluid-fluid interaction. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, pages 237–244, 2005

    2005

  61. [70]

    Phyrecon: Physically plausible neural scene reconstruction

    Junfeng Ni, Yixin Chen, Bohan Jing, Nan Jiang, Bin Wang, Bo Dai, Yixin Zhu, Song-Chun Zhu, and Siyuan Huang. Phyrecon: Physically plausible neural scene reconstruction. arXiv preprint arXiv:2404.16666, 2024

    2024

  62. [71]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR, 2021

    2021

  63. [72]

    Graphical modeling and animation of ductile fracture

    James F O’brien, Adam W Bargteil, and Jessica K Hodgins. Graphical modeling and animation of ductile fracture. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pages 291–294, 2002

    2002

  64. [73]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023a, 2023

    2023 arXiv

  65. [74]

    Gpt-4v(ision) system card, 2023b

    OpenAI. Gpt-4v(ision) system card, 2023b. https://openai.com/research/ gpt-4v-system-card , 2023

    2023

  66. [75]

    GitHub - hpcaitech/Open-Sora: Open-Sora: Democratizing Efficient Video Pro- duction for All — github.com

    OpenSora. GitHub - hpcaitech/Open-Sora: Open-Sora: Democratizing Efficient Video Pro- duction for All — github.com. https://github.com/hpcaitech/Open-Sora, 2024

    2024

  67. [76]

    Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models

    Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, et al. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models. arXiv preprint arXiv:2405.02287, 2024

    2024

  68. [77]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  69. [78]

    Pika — pika.art

    pika. Pika — pika.art. https://pika.art/

  70. [79]

    Intuitive physics learning in a deep-learning model inspired by developmental psychology

    Luis S Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. Intuitive physics learning in a deep-learning model inspired by developmental psychology. Nature human behaviour, 6(9):1257–1267, 2022

    2022

  71. [80]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  72. [81]

    Power plastics: A hybrid lagrangian/eulerian solver for mesoscale inelastic flows

    Ziyin Qu, Minchen Li, Yin Yang, Chenfanfu Jiang, and Fernando De Goes. Power plastics: A hybrid lagrangian/eulerian solver for mesoscale inelastic flows. ACM Transactions on Graphics (TOG), 42(6):1–11, 2023

    2023

  73. [82]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  74. [83]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  75. [84]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv pr...

    2024 arXiv

  76. [86]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  77. [87]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  78. [88]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  79. [89]

    Physcap: Physically plausible monocular 3d motion capture in real time

    Soshi Shimada, Vladislav Golyanik, Weipeng Xu, and Christian Theobalt. Physcap: Physically plausible monocular 3d motion capture in real time. ACM Transactions on Graphics (ToG), 39(6):1–16, 2020

    2020

  80. [90]

    Fem simulation of 3d deformable solids: a practitioner’s guide to theory, discretization and model reduction

    Eftychios Sifakis and Jernej Barbic. Fem simulation of 3d deformable solids: a practitioner’s guide to theory, discretization and model reduction. In Acm siggraph 2012 courses, pages 1–50. 2012

    2012

  81. [91]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022

    2022 arXiv

  82. [92]

    Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2

    Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3626–3636, 2022

    2022

  83. [93]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  84. [94]

    A material point method for snow simulation.ACM Transactions on Graphics (TOG), 32(4):1–10, 2013

    Alexey Stomakhin, Craig Schroeder, Lawrence Chai, Joseph Teran, and Andrew Selle. A material point method for snow simulation.ACM Transactions on Graphics (TOG), 32(4):1–10, 2013

    2013

  85. [95]

    Journeydb: A benchmark for generative image understanding

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36, 2024

    2024

  86. [96]

    Fast simulation of viscous fluids with elasticity and thermal conductivity using position-based dynamics

    Tetsuya Takahashi, Tomoyuki Nishita, and Issei Fujishiro. Fast simulation of viscous fluids with elasticity and thermal conductivity using position-based dynamics. Computers & Graphics, 43:21–30, 2014

    2014

  87. [97]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020

    2020

  88. [98]

    Winoground: Probing vision and language models for visio-linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2022

  89. [99]

    Towards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michal- ski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018

    2018 arXiv

  90. [100]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Lear...

    2022

  91. [101]

    Diffusion model alignment using direct preference optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint arXiv:2311.12908, 2023

    2023

  92. [102]

    Survey of bias in text-to-image generation: Definition, evaluation, and mitigation

    Yixin Wan, Arjun Subramonian, Anaelia Ovalle, Zongyu Lin, Ashima Suvarna, Christina Chance, Hritik Bansal, Rebecca Pattichis, and Kai-Wei Chang. Survey of bias in text-to-image generation: Definition, evaluation, and mitigation. arXiv preprint arXiv:2404.01030, 2024

    2024

  93. [103]

    Modelscope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023

    2023 arXiv

  94. [104]

    Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation

    Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. arXiv preprint arXiv:2305.10874, 2023

    2023

  95. [105]

    Lavie: High-quality video generation with cascaded latent diffusion models

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103, 2023

    2023

  96. [106]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023

    2023 arXiv

  97. [107]

    Non-newtonian behaviour of heather honey

    Mariusz Witczak, Lesław Juszczak, and Dorota Gałkowska. Non-newtonian behaviour of heather honey. Journal of Food Engineering, 104(4):532–537, 2011

    2011

  98. [108]

    Anisompm: Animating anisotropic damage mechanics: Supplemental document

    Joshuah Wolper, Yunuo Chen, Minchen Li, Yu Fang, Ziyin Qu, Jiecong Lu, Meggie Cheng, and Chenfanfu Jiang. Anisompm: Animating anisotropic damage mechanics: Supplemental document. ACM Trans. Graph, 39(4), 2020

    2020

  99. [109]

    A contact proxy splitting method for lagrangian solid-fluid coupling

    Tianyi Xie, Minchen Li, Yin Yang, and Chenfanfu Jiang. A contact proxy splitting method for lagrangian solid-fluid coupling. ACM Transactions on Graphics (TOG), 42(4):1–14, 2023

    2023

  100. [110]

    Physgaussian: Physics-integrated 3d gaussians for generative dynamics

    Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. arXiv preprint arXiv:2311.12198, 2023

    2023

  101. [111]

    Interdiff: Generating 3d human-object interactions with physics-informed diffusion

    Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14928–14940, 2023

    2023

  102. [112]

    Advancing high-resolution video-language representation with large-scale video transcriptions

    Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2022

  103. [113]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  104. [114]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023

    2023 arXiv

  105. [115]

    Perception of 3d shape integrates intuitive physics and analysis-by-synthesis

    Ilker Yildirim, Max H Siegel, Amir A Soltani, Shraman Ray Chaudhuri, and Joshua B Tenenbaum. Perception of 3d shape integrates intuitive physics and analysis-by-synthesis. Nature Human Behaviour, 8(2):320–335, 2024

    2024

  106. [116]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023

    2023 arXiv

  107. [117]

    Physdiff: Physics-guided human motion diffusion model

    Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16010–16021, 2023. 18

    2023

  108. [118]

    Continuum foam: A material point method for shear-dependent flows

    Yonghao Yue, Breannan Smith, Christopher Batty, Changxi Zheng, and Eitan Grinspun. Continuum foam: A material point method for shear-dependent flows. ACM Transactions on Graphics (TOG), 34(5):1–20, 2015

    2015

  109. [119]

    Physdreamer: Physics-based interaction with 3d objects via video generation

    Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. arXiv preprint arXiv:2404.13026, 2024

    2024

  110. [120]

    Not all tasks are born equal: Understanding zero-shot generalization

    Jing Zhou, Zongyu Lin, Yanan Zheng, Jian Li, and Zhilin Yang. Not all tasks are born equal: Understanding zero-shot generalization. In The Eleventh International Conference on Learning Representations, 2022

    2022

  111. [121]

    A convex formulation of frictional contact for the material point method and rigid bodies

    Zeshun Zong, Chenfanfu Jiang, and Xuchen Han. A convex formulation of frictional contact for the material point method and rigid bodies. arXiv preprint arXiv:2403.13783, 2024

    2024

  112. [122]

    Neural stress fields for reduced-order elastoplasticity and fracture

    Zeshun Zong, Xuan Li, Minchen Li, Maurizio M Chiaramonte, Wojciech Matusik, Eitan Grinspun, Kevin Carlberg, Chenfanfu Jiang, and Peter Yichen Chen. Neural stress fields for reduced-order elastoplasticity and fracture. In SIGGRAPH Asia 2023 Conference Papers, pages 1–11, 2023. ...

    2023

  113. [123]

    from Google research. Specifically, Lumiere-T2V generates a video conditioned on the text, while Lumiere-T2I2V generates a video conditioned on an image, that is in-turn generated with the caption using a text-to-image generative model [87]. CogVideoX [113] is a most recent op...

  114. [124]

    Focus on common solids used in everyday scenarios, avoiding rare or seldom-used materials

  115. [126]

    Avoid generating static scenes (e.g., ‘Lid covers pot to retain heat’, ‘Stack of paper sits on the desk’)

  116. [129]

    ## Scratching action that requires penetration A key scratches the surface of a wooden table

    Format each caption as follows: ‘action’: ACTION, ‘solid 1’: SOLID, ‘solid 2’: SOLID, ‘caption’: CAPTION Bad Examples Of Captions (Do Not Generate Such Captions): A diamond scratching glass. ## Scratching action that requires penetration A key scratches the surface of a wooden...

  117. [130]

    Focus on common solids and fluids used in everyday scenarios, avoiding rare or seldom-used materials

  118. [132]

    Avoid actions that execute state change from solid to fluid or vice-versa

  119. [135]

    Do not focus on the actions that require penetration forces

    The captions should focus on the actions that require contact forces, or friction forces. Do not focus on the actions that require penetration forces

  120. [136]

    ## dissolving action will not be visible in video Sulfuric acid corroding metal

    Format each caption as follows: ‘action’: ACTION, ‘solid’: SOLID, ‘fluid’: FLUID, ‘caption’: CAPTION Bad Examples Of Captions (Do Not Generate Such Captions): Sugar dissolves in water. ## dissolving action will not be visible in video Sulfuric acid corroding metal. ## corrosio...

  121. [137]

    Focus on common fluids used in everyday scenarios, avoiding rare or seldom-used materials

  122. [138]

    Exclude actions like ‘celebrating’, ‘arguing’, or ‘laughing’ that do not clearly involve physical interaction between materials

  123. [139]

    Avoid generating static scenes (e.g., ‘Lid covers pot to retain heat’)

  124. [140]

    Avoid adding participle phrases (e.g., ‘sweetening it’, ‘a creamy swirl’, ‘fizzing energetically’) in the caption

  125. [141]

    The captions should focus on the actions that require mixing and laying for liquid-liquid interactions, or some contact forces between liquid and gas

  126. [142]

    ## solidification won’t be visible in the video Sugar disappears into stirring water

    Format each caption as follows: ‘action’: ACTION, ‘fluid 1’: FLUID, ‘fluid 2’: FLUID, ‘caption’: CAPTION Bad Examples Of Captions (Do Not Generate Such Captions): Juice solidifies around water in ice trays. ## solidification won’t be visible in the video Sugar disappears into ...

  127. [143]

    Subsequently, we calculate the pearson correlation between video quality and motion with physical commonsense and semantic adherence

    and video motion using the RAFT optical flow model [ 97]. Subsequently, we calculate the pearson correlation between video quality and motion with physical commonsense and semantic adherence. We present the results in Table 11. We find that physical commonsense and semantic ad...

Pith tools

Reviewed May 20, 2026 · model on record in the stance chip above.