{"work":{"id":"77fd5ac9-ae98-4846-9d83-e9c73c8f2a52","openalex_id":null,"doi":null,"arxiv_id":"2405.21075","raw_key":null,"title":"Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis","authors":null,"authors_text":"Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang","year":2024,"venue":"cs.CV","abstract":"In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs in processing sequential visual data is still insufficiently explored, highlighting the absence of a comprehensive, high-quality assessment of their performance. In this paper, we introduce Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis. Our work distinguishes from existing benchmarks through four key features: 1) Diversity in video types, spanning 6 primary visual domains with 30 subfields to ensure broad scenario generalizability; 2) Duration in temporal dimension, encompassing both short-, medium-, and long-term videos, ranging from 11 seconds to 1 hour, for robust contextual dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides video frames, including subtitles and audios, to unveil the all-round capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual labeling by expert annotators to facilitate precise and reliable model assessment. 900 videos with a total of 254 hours are manually selected and annotated by repeatedly viewing all the video content, resulting in 2,700 question-answer pairs. With Video-MME, we extensively evaluate various state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as open-source image models like InternVL-Chat-V1.5 and video models like LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the best-performing commercial model, significantly outperforming the open-source models. Our dataset along with these findings underscores the need for further improvements in handling longer sequences and multi-modal data. Project Page: https://video-mme.github.io","external_url":"https://arxiv.org/abs/2405.21075","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-05-23T17:33:15.675304+00:00","pith_arxiv_id":"2405.21075","created_at":"2026-05-09T05:55:31.695067+00:00","updated_at":"2026-05-23T17:33:15.675304+00:00","title_quality_ok":true,"display_title":"Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis","render_title":"Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis"},"hub":{"state":{"work_id":"77fd5ac9-ae98-4846-9d83-e9c73c8f2a52","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":88,"external_cited_by_count":null,"distinct_field_count":7,"first_pith_cited_at":"2024-02-13T07:47:36+00:00","last_pith_cited_at":"2026-05-21T08:27:15+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-05-23T19:04:31.052369+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"dataset","n":17},{"context_role":"background","n":11},{"context_role":"baseline","n":1}],"polarity_counts":[{"context_polarity":"use_dataset","n":16},{"context_polarity":"background","n":11},{"context_polarity":"baseline","n":1},{"context_polarity":"unclear","n":1}],"runs":{"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-14T17:59:53.987653+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"Qwen2.5-VL Technical Report","work_id":"69dffacb-bfe8-442d-be86-48624c60426f","shared_citers":17},{"title":"Mlvu: A comprehensive benchmark for multi-task long video understanding","work_id":"346256da-dd21-4cc3-9a98-519467614854","shared_citers":16},{"title":"LLaVA-OneVision: Easy Visual Task Transfer","work_id":"f5f2452b-f2a9-49ac-b38d-c76e18cdfe49","shared_citers":15},{"title":"Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution","work_id":"8abcfe4f-e0fb-44b7-9123-448fac95f90a","shared_citers":14},{"title":"Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context","work_id":"80e3e977-f1bb-4c83-8d0c-1ab0a0c5c3f1","shared_citers":12},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":12},{"title":"Are We on the Right Way for Evaluating Large Vision-Language Models?","work_id":"0d0b977c-a42e-49b1-869e-b7360dca5282","shared_citers":11},{"title":"Gemini: A Family of Highly Capable Multimodal Models","work_id":"83f7c85b-3f11-450f-ac0c-64d9745220b2","shared_citers":11},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":11},{"title":"MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark","work_id":"19883673-604e-4b58-b036-5a04ec11a6f9","shared_citers":11},{"title":"Long Context Transfer from Language to Vision","work_id":"52f1b946-568f-4819-9d8a-a87296f8852d","shared_citers":10},{"title":"MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts","work_id":"e22c3789-9e71-4242-b6ea-3e60e06e2b66","shared_citers":10},{"title":"VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs","work_id":"ccfc3f89-c510-45f1-8a35-ed1a56c0ae5c","shared_citers":10},{"title":"LLaVA-Video: Video Instruction Tuning With Synthetic Data","work_id":"e598f516-d992-449a-ab6d-6c788b3a1d7b","shared_citers":9},{"title":"Longvideobench: A benchmark for long-context inter- leaved video-language understanding","work_id":"f6bb6dd9-35f5-4788-9063-af4d76699147","shared_citers":9},{"title":"Blink: Multimodal large language models can see but not perceive","work_id":"10e71374-f61b-477c-9433-12b7990c388e","shared_citers":8},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":8},{"title":"Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling","work_id":"ee70bdc8-4656-4849-ada7-ce42a2278d70","shared_citers":7},{"title":"How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites","work_id":"3714835e-c5a6-4d7e-950c-be44670ed9e6","shared_citers":7},{"title":"Lvbench: An extreme long video understanding benchmark","work_id":"e9bfdf40-cd28-4d57-98b8-31b7e97f9f2f","shared_citers":7},{"title":"Measuring multimodal math- ematical reasoning with math-vision dataset","work_id":"c59c0707-68e6-4ab4-9b9f-293004398dc7","shared_citers":7},{"title":"Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi","work_id":"da087b16-ea05-4064-980e-ce1d6e281d49","shared_citers":7},{"title":"Qwen2 Technical Report","work_id":"a1857881-ab9b-4b80-9b5f-9ae4b5c2566d","shared_citers":7},{"title":"Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi","work_id":"b6841af7-2b88-4597-9d0e-54e96aa7a368","shared_citers":7}],"time_series":[{"n":5,"year":2024},{"n":12,"year":2025},{"n":21,"year":2026}],"dependency_candidates":[]},"error":null,"updated_at":"2026-05-14T17:59:23.560143+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"items":[{"title":"Qwen3 Technical Report","outcome":"unchanged","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"counts":{"fixed":0,"merged":0,"unchanged":1,"quarantined":0,"needs_external_resolution":0},"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-14T17:59:41.407087+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis","claims":[{"claim_text":"In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs in processing sequential visual data is still insufficiently explored, highlighting the absence of a comprehensive, high-quality assessment of their performance. In this paper, we introduce Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis. Our work distinguishes from existing","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-14T17:59:32.282698+00:00"}},"summary":{"title":"Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis","claims":[{"claim_text":"In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs in processing sequential visual data is still insufficiently explored, highlighting the absence of a comprehensive, high-quality assessment of their performance. In this paper, we introduce Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis. Our work distinguishes from existing","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis because it crossed a citation-hub threshold.","role_counts":[]},"graph":{"co_cited":[{"title":"Qwen2.5-VL Technical Report","work_id":"69dffacb-bfe8-442d-be86-48624c60426f","shared_citers":17},{"title":"Mlvu: A comprehensive benchmark for multi-task long video understanding","work_id":"346256da-dd21-4cc3-9a98-519467614854","shared_citers":16},{"title":"LLaVA-OneVision: Easy Visual Task Transfer","work_id":"f5f2452b-f2a9-49ac-b38d-c76e18cdfe49","shared_citers":15},{"title":"Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution","work_id":"8abcfe4f-e0fb-44b7-9123-448fac95f90a","shared_citers":14},{"title":"Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context","work_id":"80e3e977-f1bb-4c83-8d0c-1ab0a0c5c3f1","shared_citers":12},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":12},{"title":"Are We on the Right Way for Evaluating Large Vision-Language Models?","work_id":"0d0b977c-a42e-49b1-869e-b7360dca5282","shared_citers":11},{"title":"Gemini: A Family of Highly Capable Multimodal Models","work_id":"83f7c85b-3f11-450f-ac0c-64d9745220b2","shared_citers":11},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":11},{"title":"MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark","work_id":"19883673-604e-4b58-b036-5a04ec11a6f9","shared_citers":11},{"title":"Long Context Transfer from Language to Vision","work_id":"52f1b946-568f-4819-9d8a-a87296f8852d","shared_citers":10},{"title":"MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts","work_id":"e22c3789-9e71-4242-b6ea-3e60e06e2b66","shared_citers":10},{"title":"VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs","work_id":"ccfc3f89-c510-45f1-8a35-ed1a56c0ae5c","shared_citers":10},{"title":"LLaVA-Video: Video Instruction Tuning With Synthetic Data","work_id":"e598f516-d992-449a-ab6d-6c788b3a1d7b","shared_citers":9},{"title":"Longvideobench: A benchmark for long-context inter- leaved video-language understanding","work_id":"f6bb6dd9-35f5-4788-9063-af4d76699147","shared_citers":9},{"title":"Blink: Multimodal large language models can see but not perceive","work_id":"10e71374-f61b-477c-9433-12b7990c388e","shared_citers":8},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":8},{"title":"Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling","work_id":"ee70bdc8-4656-4849-ada7-ce42a2278d70","shared_citers":7},{"title":"How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites","work_id":"3714835e-c5a6-4d7e-950c-be44670ed9e6","shared_citers":7},{"title":"Lvbench: An extreme long video understanding benchmark","work_id":"e9bfdf40-cd28-4d57-98b8-31b7e97f9f2f","shared_citers":7},{"title":"Measuring multimodal math- ematical reasoning with math-vision dataset","work_id":"c59c0707-68e6-4ab4-9b9f-293004398dc7","shared_citers":7},{"title":"Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi","work_id":"da087b16-ea05-4064-980e-ce1d6e281d49","shared_citers":7},{"title":"Qwen2 Technical Report","work_id":"a1857881-ab9b-4b80-9b5f-9ae4b5c2566d","shared_citers":7},{"title":"Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi","work_id":"b6841af7-2b88-4597-9d0e-54e96aa7a368","shared_citers":7}],"time_series":[{"n":5,"year":2024},{"n":12,"year":2025},{"n":21,"year":2026}],"dependency_candidates":[]},"authors":[]}}