{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2023:XCXLFWSANHUSBZ2RU7WQZVMB2J","short_pith_number":"pith:XCXLFWSA","schema_version":"1.0","canonical_sha256":"b8aeb2da4069e920e751a7ed0cd581d277d482abcbec9b58570126d1319f18bf","source":{"kind":"arxiv","id":"2312.14125","version":4},"attestation_state":"computed","paper":{"title":"VideoPoet: A Large Language Model for Zero-Shot Video Generation","license":"http://creativecommons.org/licenses/by/4.0/","headline":"A decoder-only transformer trained on multimodal data generates high-fidelity videos zero-shot from text, images, and audio.","cross_cats":["cs.AI"],"primary_cat":"cs.CV","authors_text":"Agrim Gupta, Alonso Martinez, Anja Hauth, Bryan Seybold, Dan Kondratyuk, David A. Ross, David Hendon, David Minnen, Grant Schindler, Hartwig Adam, Hassan Akbari, Huisheng Wang, Irfan Essa, Jimmy Yan, Jonathan Huang, Jos\\'e Lezama, Josh Dillon, Kihyuk Sohn, Krishna Somandepalli, Lijun Yu, Lu Jiang, Meera Hahn, Mikhail Sirotenko, Ming-Chang Chiu, Ming-Hsuan Yang, Rachel Hornung, Vighnesh Birodkar, Xiuye Gu, Xuan Yang, Yair Alon, Yong Cheng","submitted_at":"2023-12-21T18:46:41Z","abstract_excerpt":"We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundatio"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":false},"canonical_record":{"source":{"id":"2312.14125","kind":"arxiv","version":4},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CV","submitted_at":"2023-12-21T18:46:41Z","cross_cats_sorted":["cs.AI"],"title_canon_sha256":"18b0d90fb9392b704e8d07576d72bd1dfbee6398ddf50fe6abad23281ad20228","abstract_canon_sha256":"544d649ec5a1c6c28686e7b7a0beb204295ee44d4bde2e405b1197873c44127f"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:50.672877Z","signature_b64":"g9cBJSGJITYhXXePuLJUBMLYOw2tn9gA2U3F+Uo32KkWWufZFKtYsnN/1u4nE5Z7RJVajM7kF+EARteh4VmCCQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"b8aeb2da4069e920e751a7ed0cd581d277d482abcbec9b58570126d1319f18bf","last_reissued_at":"2026-05-17T23:38:50.672005Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:50.672005Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"VideoPoet: A Large Language Model for Zero-Shot Video Generation","license":"http://creativecommons.org/licenses/by/4.0/","headline":"A decoder-only transformer trained on multimodal data generates high-fidelity videos zero-shot from text, images, and audio.","cross_cats":["cs.AI"],"primary_cat":"cs.CV","authors_text":"Agrim Gupta, Alonso Martinez, Anja Hauth, Bryan Seybold, Dan Kondratyuk, David A. Ross, David Hendon, David Minnen, Grant Schindler, Hartwig Adam, Hassan Akbari, Huisheng Wang, Irfan Essa, Jimmy Yan, Jonathan Huang, Jos\\'e Lezama, Josh Dillon, Kihyuk Sohn, Krishna Somandepalli, Lijun Yu, Lu Jiang, Meera Hahn, Mikhail Sirotenko, Ming-Chang Chiu, Ming-Hsuan Yang, Rachel Hornung, Vighnesh Birodkar, Xiuye Gu, Xuan Yang, Yair Alon, Yong Cheng","submitted_at":"2023-12-21T18:46:41Z","abstract_excerpt":"We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundatio"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio -- and demonstrates state-of-the-art capabilities in zero-shot video generation, specifically highlighting VideoPoet's ability to generate high-fidelity motions.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The assumption that a mixture of multimodal generative objectives trained autoregressively on a decoder-only transformer will transfer effectively to zero-shot video generation without requiring substantial additional video-specific inductive biases or post-training.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"VideoPoet is a large language model that performs zero-shot video generation with audio from diverse multimodal conditioning signals.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"A decoder-only transformer trained on multimodal data generates high-fidelity videos zero-shot from text, images, and audio.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"a15f98adaa3ec8f737e33d7350aec97e9be28a7d090d4bf25983cb7d7e7f9f13"},"source":{"id":"2312.14125","kind":"arxiv","version":4},"verdict":{"id":"0470c117-63d1-4cce-8ae5-5f929abddc25","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T17:46:28.066693Z","strongest_claim":"VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio -- and demonstrates state-of-the-art capabilities in zero-shot video generation, specifically highlighting VideoPoet's ability to generate high-fidelity motions.","one_line_summary":"VideoPoet is a large language model that performs zero-shot video generation with audio from diverse multimodal conditioning signals.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The assumption that a mixture of multimodal generative objectives trained autoregressively on a decoder-only transformer will transfer effectively to zero-shot video generation without requiring substantial additional video-specific inductive biases or post-training.","pith_extraction_headline":"A decoder-only transformer trained on multimodal data generates high-fidelity videos zero-shot from text, images, and audio."},"references":{"count":55,"sample":[{"doi":"","year":null,"title":"MusicLM: Generating Music From Text","work_id":"15e6566e-1c36-468f-966e-823248cbf87f","ref_index":1,"cited_arxiv_id":"2301.11325","is_internal_anchor":true},{"doi":"","year":null,"title":"Alternating gradient descent and mixture-of- experts for integrated multimodal perception","work_id":"c82d289d-9341-4a52-8917-f972c5d5da2a","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"PaLM 2 Technical Report","work_id":"905ee9a7-ea61-4a94-bd62-2600cbe3e315","ref_index":3,"cited_arxiv_id":"2305.10403","is_internal_anchor":true},{"doi":"","year":null,"title":"Lumiere: A space-time diffusion model for video generation","work_id":"8a0a0735-d82c-4090-a039-697d06ccc3f0","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets","work_id":"4f68eada-27e3-437a-a2fe-6e4ca524d0d3","ref_index":5,"cited_arxiv_id":"2311.15127","is_internal_anchor":true}],"resolved_work":55,"snapshot_sha256":"7da2eb9360b22bfbeb58e0bddb321f751c71359f68ccd8c56f9b73111db36026","internal_anchors":29},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2312.14125","created_at":"2026-05-17T23:38:50.672125+00:00"},{"alias_kind":"arxiv_version","alias_value":"2312.14125v4","created_at":"2026-05-17T23:38:50.672125+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2312.14125","created_at":"2026-05-17T23:38:50.672125+00:00"},{"alias_kind":"pith_short_12","alias_value":"XCXLFWSANHUS","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"XCXLFWSANHUSBZ2R","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"XCXLFWSA","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":29,"internal_anchor_count":29,"sample":[{"citing_arxiv_id":"2605.15391","citing_title":"PanoWorld: Geometry-Consistent Panoramic Video World Modeling","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2507.13942","citing_title":"Frozen Forecasting: A Unified Evaluation","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2509.12052","citing_title":"FluentAvatar: Flicker-Free Talking-Head Animation via Phoneme-Guided Autoregressive Modeling","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2509.23980","citing_title":"Towards Redundancy Reduction in Diffusion Models for Efficient Video Super-Resolution","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2412.14169","citing_title":"Autoregressive Video Generation without Vector Quantization","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2512.00336","citing_title":"MVAD: A Benchmark Dataset for Multimodal AI-Generated Video-Audio Detection","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2503.19325","citing_title":"Long-Context Autoregressive Video Modeling with Next-Frame Prediction","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2509.25161","citing_title":"Rolling Forcing: Autoregressive Long Video Diffusion in Real Time","ref_index":75,"is_internal_anchor":true},{"citing_arxiv_id":"2602.07775","citing_title":"Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusion","ref_index":48,"is_internal_anchor":true},{"citing_arxiv_id":"2503.22020","citing_title":"CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models","ref_index":31,"is_internal_anchor":true},{"citing_arxiv_id":"2503.12605","citing_title":"Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey","ref_index":209,"is_internal_anchor":true},{"citing_arxiv_id":"2505.15809","citing_title":"MMaDA: Multimodal Large Diffusion Language Models","ref_index":99,"is_internal_anchor":true},{"citing_arxiv_id":"2603.09721","citing_title":"FrameDiT: Diffusion Transformer with Matrix Attention for Efficient Video Generation","ref_index":23,"is_internal_anchor":true},{"citing_arxiv_id":"2409.16283","citing_title":"Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation","ref_index":20,"is_internal_anchor":true},{"citing_arxiv_id":"2605.13178","citing_title":"CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Models","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2505.13211","citing_title":"MAGI-1: Autoregressive Video Generation at Scale","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2402.17177","citing_title":"Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models","ref_index":199,"is_internal_anchor":true},{"citing_arxiv_id":"2404.02101","citing_title":"CameraCtrl: Enabling Camera Control for Text-to-Video Generation","ref_index":127,"is_internal_anchor":true},{"citing_arxiv_id":"2506.15564","citing_title":"Show-o2: Improved Native Unified Multimodal Models","ref_index":52,"is_internal_anchor":true},{"citing_arxiv_id":"2408.12528","citing_title":"Show-o: One Single Transformer to Unify Multimodal Understanding and Generation","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2410.13720","citing_title":"Movie Gen: A Cast of Media Foundation Models","ref_index":36,"is_internal_anchor":true},{"citing_arxiv_id":"2409.18869","citing_title":"Emu3: Next-Token Prediction is All You Need","ref_index":38,"is_internal_anchor":true},{"citing_arxiv_id":"2604.16479","citing_title":"Latent-Compressed Variational Autoencoder for Video Diffusion Models","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2406.07476","citing_title":"VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2604.04502","citing_title":"Veo-Act: How Far Can Frontier Video Models Advance Generalizable Robot Manipulation?","ref_index":30,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J","json":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J.json","graph_json":"https://pith.science/api/pith-number/XCXLFWSANHUSBZ2RU7WQZVMB2J/graph.json","events_json":"https://pith.science/api/pith-number/XCXLFWSANHUSBZ2RU7WQZVMB2J/events.json","paper":"https://pith.science/paper/XCXLFWSA"},"agent_actions":{"view_html":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J","download_json":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J.json","view_paper":"https://pith.science/paper/XCXLFWSA","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2312.14125&json=true","fetch_graph":"https://pith.science/api/pith-number/XCXLFWSANHUSBZ2RU7WQZVMB2J/graph.json","fetch_events":"https://pith.science/api/pith-number/XCXLFWSANHUSBZ2RU7WQZVMB2J/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J/action/timestamp_anchor","attest_storage":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J/action/storage_attestation","attest_author":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J/action/author_attestation","sign_citation":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J/action/citation_signature","submit_replication":"https://pith.science/pith/XCXLFWSANHUSBZ2RU7WQZVMB2J/action/replication_record"}},"created_at":"2026-05-17T23:38:50.672125+00:00","updated_at":"2026-05-17T23:38:50.672125+00:00"}