{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2025:B67AW5CHREYTXE5IWIWTEKXGAE","short_pith_number":"pith:B67AW5CH","schema_version":"1.0","canonical_sha256":"0fbe0b744789313b93a8b22d322ae6012ded139e34f5e623b3294fc1413289ad","source":{"kind":"arxiv","id":"2502.04326","version":3},"attestation_state":"computed","paper":{"title":"WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs","license":"http://creativecommons.org/licenses/by/4.0/","headline":"The WorldSense benchmark shows that current multimodal models reach at most 65.1 percent accuracy on tasks requiring tight audio-visual synergy in real-world videos.","cross_cats":["cs.AI"],"primary_cat":"cs.CV","authors_text":"Jack Hong, Jiayin Cai, Shilin Yan, Weidi Xie, Xiaolong Jiang, Yao Hu","submitted_at":"2025-02-06T18:59:40Z","abstract_excerpt":"We introduce WorldSense, the first benchmark to assess the multi-modal video understanding, that simultaneously encompasses visual, audio, and text inputs. In contrast to existing benchmarks, our WorldSense has several features: (i)collaboration of omni-modality, we design the evaluation tasks to feature a strong coupling of audio and video, requiring models to effectively utilize the synergistic perception of omni-modality; (ii)diversity of videos and tasks, WorldSense encompasses a diverse collection of 1,662 audio-visual synchronised videos, systematically categorized into 8 primary domains"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2502.04326","kind":"arxiv","version":3},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CV","submitted_at":"2025-02-06T18:59:40Z","cross_cats_sorted":["cs.AI"],"title_canon_sha256":"0ed6a53c364bbeb8baa039d1b28da43b1abbf6d28b95c5ed889d1e5a8e88afe4","abstract_canon_sha256":"e6e693f0a6a7a9ed5e057f30399701ce9984f76b3b747e47a28b490fd1a84d67"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:14.897310Z","signature_b64":"raRYUF3r/XddOKRPPWfst3VKWDgR6n0s37BF3M3zXGo6SSt/hVWCbHo2E9shuOWvZT+Zh6ux+xa/h+U/cdkEBQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"0fbe0b744789313b93a8b22d322ae6012ded139e34f5e623b3294fc1413289ad","last_reissued_at":"2026-05-17T23:38:14.896721Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:14.896721Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs","license":"http://creativecommons.org/licenses/by/4.0/","headline":"The WorldSense benchmark shows that current multimodal models reach at most 65.1 percent accuracy on tasks requiring tight audio-visual synergy in real-world videos.","cross_cats":["cs.AI"],"primary_cat":"cs.CV","authors_text":"Jack Hong, Jiayin Cai, Shilin Yan, Weidi Xie, Xiaolong Jiang, Yao Hu","submitted_at":"2025-02-06T18:59:40Z","abstract_excerpt":"We introduce WorldSense, the first benchmark to assess the multi-modal video understanding, that simultaneously encompasses visual, audio, and text inputs. In contrast to existing benchmarks, our WorldSense has several features: (i)collaboration of omni-modality, we design the evaluation tasks to feature a strong coupling of audio and video, requiring models to effectively utilize the synergistic perception of omni-modality; (ii)diversity of videos and tasks, WorldSense encompasses a diverse collection of 1,662 audio-visual synchronised videos, systematically categorized into 8 primary domains"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"The experimental results indicate that existing models face significant challenges in understanding real-world scenarios (65.1% best accuracy).","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the manually annotated QA pairs and the chosen 26 tasks accurately capture the requirements of real-world omnimodal understanding without introducing annotation bias or task selection that favors certain model architectures.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"WorldSense provides the first benchmark requiring synergistic audio-video-text understanding on 1,662 real-world videos and 3,172 QA pairs, where the best current multimodal LLM reaches only 65.1% accuracy.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"The WorldSense benchmark shows that current multimodal models reach at most 65.1 percent accuracy on tasks requiring tight audio-visual synergy in real-world videos.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"50adba971c83e578b0db0f61842488817d112d8f728ff1225605f288b2be8517"},"source":{"id":"2502.04326","kind":"arxiv","version":3},"verdict":{"id":"d0ba289f-76ac-403e-85e1-c767a36b905d","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-17T05:48:39.250838Z","strongest_claim":"The experimental results indicate that existing models face significant challenges in understanding real-world scenarios (65.1% best accuracy).","one_line_summary":"WorldSense provides the first benchmark requiring synergistic audio-video-text understanding on 1,662 real-world videos and 3,172 QA pairs, where the best current multimodal LLM reaches only 65.1% accuracy.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the manually annotated QA pairs and the chosen 26 tasks accurately capture the requirements of real-world omnimodal understanding without introducing annotation bias or task selection that favors certain model architectures.","pith_extraction_headline":"The WorldSense benchmark shows that current multimodal models reach at most 65.1 percent accuracy on tasks requiring tight audio-visual synergy in real-world videos."},"references":{"count":88,"sample":[{"doi":"","year":2022,"title":"Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736","work_id":"4710bba6-9cf4-4a0c-92c0-404f40a04621","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Introducing the next generation of Claude","work_id":"1cbb4b5a-f4fe-41dd-8d13-af2db291d5b4","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Hourvideo: 1-hour video-language understanding","work_id":"8a5f3847-96f0-4e8c-aa50-0efe6ccf9dab","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Driving with llms: Fusing object-level vector modality for explainable autonomous driving","work_id":"2ee4fb29-d694-421f-9467-3ec2cf9736aa","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling","work_id":"ee70bdc8-4656-4849-ada7-ce42a2278d70","ref_index":5,"cited_arxiv_id":"2412.05271","is_internal_anchor":true}],"resolved_work":88,"snapshot_sha256":"f558c84e3e56ba3ed7577460fc9ca4468d4814064c932ef23b7df9bc6bf693de","internal_anchors":33},"formal_canon":{"evidence_count":2,"snapshot_sha256":"171201a3c31ca9854d1611ed2bbf6311253676993b22fe2fbb7f8c1a8f300d3d"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2502.04326","created_at":"2026-05-17T23:38:14.896817+00:00"},{"alias_kind":"arxiv_version","alias_value":"2502.04326v3","created_at":"2026-05-17T23:38:14.896817+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2502.04326","created_at":"2026-05-17T23:38:14.896817+00:00"},{"alias_kind":"pith_short_12","alias_value":"B67AW5CHREYT","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"B67AW5CHREYTXE5I","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"B67AW5CH","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":28,"internal_anchor_count":28,"sample":[{"citing_arxiv_id":"2605.06094","citing_title":"VISD: Enhancing Video Reasoning via Structured Self-Distillation","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2605.22907","citing_title":"VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2605.22012","citing_title":"LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16403","citing_title":"When Vision Speaks for Sound","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17360","citing_title":"Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.19660","citing_title":"OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2510.15148","citing_title":"XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2511.14582","citing_title":"OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2512.02231","citing_title":"See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2512.13281","citing_title":"VideoASMR-Bench: Can AI-Generated ASMR Videos Fool VLMs and Humans?","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2511.05271","citing_title":"DeepEyesV2: Toward Agentic Multimodal Model","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2603.18856","citing_title":"Motion-o: Trajectory-Grounded Video Reasoning","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12034","citing_title":"Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2604.02605","citing_title":"Do Audio-Visual Large Language Models Really See and Hear?","ref_index":23,"is_internal_anchor":true},{"citing_arxiv_id":"2504.05299","citing_title":"SmolVLM: Redefining small and efficient multimodal models","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2604.02934","citing_title":"PolyReal: A Benchmark for Real-World Polymer Science Workflows","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2604.03995","citing_title":"A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoning","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12056","citing_title":"OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12034","citing_title":"Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2604.27393","citing_title":"MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction","ref_index":80,"is_internal_anchor":true},{"citing_arxiv_id":"2605.06094","citing_title":"VISD: Enhancing Video Reasoning via Structured Self-Distillation","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2605.06094","citing_title":"VISD: Enhancing Video Reasoning via Structured Self-Distillation","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2604.11244","citing_title":"Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2604.11627","citing_title":"POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs","ref_index":27,"is_internal_anchor":true},{"citing_arxiv_id":"2605.07593","citing_title":"TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos","ref_index":35,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/B67AW5CHREYTXE5IWIWTEKXGAE","json":"https://pith.science/pith/B67AW5CHREYTXE5IWIWTEKXGAE.json","graph_json":"https://pith.science/api/pith-number/B67AW5CHREYTXE5IWIWTEKXGAE/graph.json","events_json":"https://pith.science/api/pith-number/B67AW5CHREYTXE5IWIWTEKXGAE/events.json","paper":"https://pith.science/paper/B67AW5CH"},"agent_actions":{"view_html":"https://pith.science/pith/B67AW5CHREYTXE5IWIWTEKXGAE","download_json":"https://pith.science/pith/B67AW5CHREYTXE5IWIWTEKXGAE.json","view_paper":"https://pith.science/paper/B67AW5CH","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2502.04326&json=true","fetch_graph":"https://pith.science/api/pith-number/B67AW5CHREYTXE5IWIWTEKXGAE/graph.json","fetch_events":"https://pith.science/api/pith-number/B67AW5CHREYTXE5IWIWTEKXGAE/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/B67AW5CHREYTXE5IWIWTEKXGAE/action/timestamp_anchor","attest_storage":"https://pith.science/pith/B67AW5CHREYTXE5IWIWTEKXGAE/action/storage_attestation","attest_author":"https://pith.science/pith/B67AW5CHREYTXE5IWIWTEKXGAE/action/author_attestation","sign_citation":"https://pith.science/pith/B67AW5CHREYTXE5IWIWTEKXGAE/action/citation_signature","submit_replication":"https://pith.science/pith/B67AW5CHREYTXE5IWIWTEKXGAE/action/replication_record"}},"created_at":"2026-05-17T23:38:14.896817+00:00","updated_at":"2026-05-17T23:38:14.896817+00:00"}