{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:XD6Y6EP7RYKT4SMQC4CJJBPCNK","short_pith_number":"pith:XD6Y6EP7","schema_version":"1.0","canonical_sha256":"b8fd8f11ff8e153e499017049485e26ab991852fb84c3b7a9514944acb09a738","source":{"kind":"arxiv","id":"2406.09411","version":2},"attestation_state":"computed","paper":{"title":"MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"MuirBench reveals that even leading multimodal LLMs like GPT-4o achieve only 68 percent accuracy on multi-image tasks.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.CV","authors_text":"Chaowei Xiao, Chunyuan Li, Dan Roth, Fei Wang, Hoifung Poon, Hsiang-Hui Liu, James Y. Huang, Kai-Wei Chang, Kai Zhang, Mingyu Derek Ma, Muhao Chen, Nan Xu, Pan Lu, Qin Liu, Sheng Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Wenxuan Zhou, Xiaogeng Liu, Xingyu Fu, Zekun Li","submitted_at":"2024-06-13T17:59:52Z","abstract_excerpt":"We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLM"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":false},"canonical_record":{"source":{"id":"2406.09411","kind":"arxiv","version":2},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.CV","submitted_at":"2024-06-13T17:59:52Z","cross_cats_sorted":["cs.AI","cs.CL"],"title_canon_sha256":"7305f5e7a2333182d246e61898bc65d3ee91cf09ceb7f4a464b2086a045c3a99","abstract_canon_sha256":"8d5d7b9915588fe1d70c0c7a1399795ba30490968aba4c01fe00fe7d034964be"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:46.019264Z","signature_b64":"5dPn+EzJsnpplRMK26ZNeBRqBkE7yI2ym0nbSEeeQpQzM853FIKzXhfA/vai2nPsn/jVD5OIGP9Y7aGuTdAUDw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"b8fd8f11ff8e153e499017049485e26ab991852fb84c3b7a9514944acb09a738","last_reissued_at":"2026-05-17T23:38:46.018797Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:46.018797Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"MuirBench reveals that even leading multimodal LLMs like GPT-4o achieve only 68 percent accuracy on multi-image tasks.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.CV","authors_text":"Chaowei Xiao, Chunyuan Li, Dan Roth, Fei Wang, Hoifung Poon, Hsiang-Hui Liu, James Y. Huang, Kai-Wei Chang, Kai Zhang, Mingyu Derek Ma, Muhao Chen, Nan Xu, Pan Lu, Qin Liu, Sheng Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Wenxuan Zhou, Xiaogeng Liu, Xingyu Fu, Zekun Li","submitted_at":"2024-06-13T17:59:52Z","abstract_excerpt":"We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLM"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The assumption that each standard instance paired with an unanswerable variant has only minimal semantic differences and that this pairing reliably isolates multi-image understanding without introducing new biases or artifacts in question construction.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"MuirBench is a new benchmark showing that top multimodal LLMs struggle with robust multi-image understanding, with GPT-4o at 68% and open-source models below 33% accuracy.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"MuirBench reveals that even leading multimodal LLMs like GPT-4o achieve only 68 percent accuracy on multi-image tasks.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"f22245b469c7ad7e6126a9d040394e74a3f5ab4909a4909f6f4b90571d7f75bc"},"source":{"id":"2406.09411","kind":"arxiv","version":2},"verdict":{"id":"2e50a032-c91a-43be-8787-a13e2fbb6ccc","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-17T01:04:18.981811Z","strongest_claim":"Even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy.","one_line_summary":"MuirBench is a new benchmark showing that top multimodal LLMs struggle with robust multi-image understanding, with GPT-4o at 68% and open-source models below 33% accuracy.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The assumption that each standard instance paired with an unanswerable variant has only minimal semantic differences and that this pairing reliably isolates multi-image understanding without introducing new biases or artifacts in question construction.","pith_extraction_headline":"MuirBench reveals that even leading multimodal LLMs like GPT-4o achieve only 68 percent accuracy on multi-image tasks."},"references":{"count":72,"sample":[{"doi":"","year":2022,"title":"Flamingo: a visual language model for few-shot learning","work_id":"40995f15-58e1-4bdb-8885-4ad729de9a28","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models","work_id":"87bfa84a-e663-4165-806f-93ef439d88d0","ref_index":2,"cited_arxiv_id":"2308.01390","is_internal_anchor":true},{"doi":"","year":2023,"title":"Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023","work_id":"17a586f0-57b0-4fa0-9575-6754e69e1eab","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2020,"title":"Visual question answering on image sets","work_id":"faa336a7-da1b-4ddd-bb2b-a6766b8099d6","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":1901,"title":"Language models are few-shot learners","work_id":"04bc68bc-b7df-4ec1-8599-da037bd4f085","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":72,"snapshot_sha256":"e1374b015159682506622692e9d5d0dee45caa86d8850ff119ba272abca798ba","internal_anchors":13},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2406.09411","created_at":"2026-05-17T23:38:46.018874+00:00"},{"alias_kind":"arxiv_version","alias_value":"2406.09411v2","created_at":"2026-05-17T23:38:46.018874+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2406.09411","created_at":"2026-05-17T23:38:46.018874+00:00"},{"alias_kind":"pith_short_12","alias_value":"XD6Y6EP7RYKT","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"XD6Y6EP7RYKT4SMQ","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"XD6Y6EP7","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":23,"internal_anchor_count":23,"sample":[{"citing_arxiv_id":"2502.13923","citing_title":"Qwen2.5-VL Technical Report","ref_index":29,"is_internal_anchor":true},{"citing_arxiv_id":"2511.14159","citing_title":"MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs","ref_index":53,"is_internal_anchor":true},{"citing_arxiv_id":"2605.20950","citing_title":"Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models","ref_index":58,"is_internal_anchor":true},{"citing_arxiv_id":"2605.19726","citing_title":"Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention","ref_index":41,"is_internal_anchor":true},{"citing_arxiv_id":"2505.16933","citing_title":"LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning","ref_index":74,"is_internal_anchor":true},{"citing_arxiv_id":"2512.05929","citing_title":"LLM Harms: A Taxonomy and Discussion","ref_index":218,"is_internal_anchor":true},{"citing_arxiv_id":"2603.04676","citing_title":"Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Tasks","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2603.20633","citing_title":"Seed1.8 Model Card: Towards Generalized Real-World Agency","ref_index":69,"is_internal_anchor":true},{"citing_arxiv_id":"2604.27389","citing_title":"COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts","ref_index":43,"is_internal_anchor":true},{"citing_arxiv_id":"2604.27393","citing_title":"MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction","ref_index":46,"is_internal_anchor":true},{"citing_arxiv_id":"2604.27389","citing_title":"COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts","ref_index":43,"is_internal_anchor":true},{"citing_arxiv_id":"2605.09883","citing_title":"The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space","ref_index":33,"is_internal_anchor":true},{"citing_arxiv_id":"2604.22498","citing_title":"CGC: Compositional Grounded Contrast for Fine-Grained Multi-Image Understanding","ref_index":46,"is_internal_anchor":true},{"citing_arxiv_id":"2604.21921","citing_title":"Context Unrolling in Omni Models","ref_index":39,"is_internal_anchor":true},{"citing_arxiv_id":"2407.07895","citing_title":"LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models","ref_index":54,"is_internal_anchor":true},{"citing_arxiv_id":"2507.01006","citing_title":"GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning","ref_index":53,"is_internal_anchor":true},{"citing_arxiv_id":"2604.06484","citing_title":"ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs","ref_index":35,"is_internal_anchor":true},{"citing_arxiv_id":"2408.03326","citing_title":"LLaVA-OneVision: Easy Visual Task Transfer","ref_index":135,"is_internal_anchor":true},{"citing_arxiv_id":"2504.10479","citing_title":"InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models","ref_index":119,"is_internal_anchor":true},{"citing_arxiv_id":"2412.05271","citing_title":"Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling","ref_index":242,"is_internal_anchor":true},{"citing_arxiv_id":"2508.18265","citing_title":"InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency","ref_index":133,"is_internal_anchor":true},{"citing_arxiv_id":"2604.18512","citing_title":"S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models","ref_index":79,"is_internal_anchor":true},{"citing_arxiv_id":"2604.18320","citing_title":"EVE: Verifiable Self-Evolution of MLLMs via Executable Visual Transformations","ref_index":42,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/XD6Y6EP7RYKT4SMQC4CJJBPCNK","json":"https://pith.science/pith/XD6Y6EP7RYKT4SMQC4CJJBPCNK.json","graph_json":"https://pith.science/api/pith-number/XD6Y6EP7RYKT4SMQC4CJJBPCNK/graph.json","events_json":"https://pith.science/api/pith-number/XD6Y6EP7RYKT4SMQC4CJJBPCNK/events.json","paper":"https://pith.science/paper/XD6Y6EP7"},"agent_actions":{"view_html":"https://pith.science/pith/XD6Y6EP7RYKT4SMQC4CJJBPCNK","download_json":"https://pith.science/pith/XD6Y6EP7RYKT4SMQC4CJJBPCNK.json","view_paper":"https://pith.science/paper/XD6Y6EP7","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2406.09411&json=true","fetch_graph":"https://pith.science/api/pith-number/XD6Y6EP7RYKT4SMQC4CJJBPCNK/graph.json","fetch_events":"https://pith.science/api/pith-number/XD6Y6EP7RYKT4SMQC4CJJBPCNK/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/XD6Y6EP7RYKT4SMQC4CJJBPCNK/action/timestamp_anchor","attest_storage":"https://pith.science/pith/XD6Y6EP7RYKT4SMQC4CJJBPCNK/action/storage_attestation","attest_author":"https://pith.science/pith/XD6Y6EP7RYKT4SMQC4CJJBPCNK/action/author_attestation","sign_citation":"https://pith.science/pith/XD6Y6EP7RYKT4SMQC4CJJBPCNK/action/citation_signature","submit_replication":"https://pith.science/pith/XD6Y6EP7RYKT4SMQC4CJJBPCNK/action/replication_record"}},"created_at":"2026-05-17T23:38:46.018874+00:00","updated_at":"2026-05-17T23:38:46.018874+00:00"}